Multi Machine AI Readiness And Issue Triage

Multi-Machine AI Readiness and Issue Triage

Use when a workspace spans multiple machines, multiple repos, scheduled jobs, and multiple AI provider subscriptions.

When to Use

• User wants all machines kept in a "ready" state
• There are cron jobs / scheduled tasks across machines
• There are GitHub issues in both a hub repo and individual repos
• User has multiple AI subscriptions and wants to maximize weekly usage
• Need to decide what to fix first across platform vs repo-local work

Core Approach

1. Identify control-plane docs first

Read these first if present:

• AGENTS.md
• docs/WORKSPACE_HUB_REPOSITORY_OVERVIEW.md
• docs/ops/scheduled-tasks.md
• config/scheduled-tasks/schedule-tasks.yaml
• docs/standards/AI_REVIEW_ROUTING_POLICY.md
• session-analysis or readiness scripts under scripts/analysis/, scripts/cron/, scripts/maintenance/

Goal: determine machine roles, scheduled automation, provider roles, and readiness signals before touching issues.

2. Separate machine roles from provider roles

Do not assume every machine needs every provider.

Model the estate like this:

• Primary machine: orchestration/control plane, full cron suite, repo sync, learning pipeline
• Secondary machine(s): contribution lanes, repo sync, selected provider/tool installs
• Licensed/specialized machines: minimal repo readiness first; AI harness only if needed for their role

3. Define readiness explicitly

A machine is "ready" only if all relevant checks pass:

• reachable
• workspace present in expected path
• repos synced / auth working
• required CLIs installed (git, uv, provider CLIs, hermes, etc.)
• scheduled jobs installed for that machine variant
• logs being written
• harness/tool versions not badly drifted

Useful evidence sources:

• scheduled-task inventory docs
• repo sync logs
• harness update logs
• tool-parity snapshots like config/ai_agents/ai-tools-status.yaml
• session-analysis / learning-report logs

4. Allocate providers by economic role, not evenly

For mixed subscriptions, do not spread tools uniformly across all machines.

Recommended pattern:

• Highest-tier provider -> orchestration, planning, cross-repo synthesis, final integration
• Mid/low-cost duplicate seats -> implementation lane + reviewer/overflow lane
• Single selective seat -> architecture review, long-context synthesis, ambiguity resolution

Default routing heuristic:

• Provider A: decide / orchestrate
• Provider B seat 1: implement
• Provider B seat 2: adversarial review / overflow
• Provider C: architecture or research-only escalation

5. Triage issues in three layers

Never treat all issues from all repos as one flat backlog.

Layer 1: Hub / ecosystem issues

Prioritize first when they affect:

• multiple machines
• multiple repos
• cron or sync health
• AI routing / harness parity
• issue/work-queue hygiene

Layer 2: Repo-specific issues

Prioritize by business value:

• client/delivery blockers
• shared/core repos used by others
• personal/support repos later

Layer 3: Umbrella/meta issues

Only prioritize early if they clearly unblock many concrete issues. Otherwise decompose and defer until operational blockers are stable.

6. Score issues with a reusable rubric

For each issue, assess:

• impact
• urgency
• frequency
• blast radius
• leverage
• effort
• dependency/unblocking value

Plain rule: do first what affects many places, happens often, blocks others, and is cheap/moderate to fix.

7. Recommended cadence

• Daily: blockers, sync/auth/tool failures, active delivery issues
• Twice weekly: hub operational issues, harness parity, machine readiness
• Weekly: repo improvements, backlog grooming, skills/harness refinement
• Monthly: umbrella issues, broad refactors, non-urgent consolidation

Practical Output Format

When reporting findings, produce:

1. Current machine roles
2. Current provider/subscription allocation
3. Readiness definition
4. Priority issue buckets:
   • Ready now
   • Needs plan
   • Needs dependencies
   • Defer
5. Recommended next-wave issues
6. Provider-to-task routing policy

Good First Targets

If the ecosystem is unstable, prioritize issues about:

• provider/tool install parity across machines
• cross-machine cron health monitoring
• review/routing policy operationalization
• issue template / machine-scope hygiene

Agent Config Parity via Repo Templates

When the ecosystem uses multiple AI agents (Codex, Codex, Gemini, Hermes, etc.), manage configs centrally:

Pattern: config/agents/{provider}/ → sync script → ~/.{agent}/

• Store non-secret config templates in config/agents/{provider}/ in the hub repo
• Use a sync script (e.g. scripts/_core/sync-agent-configs.sh) that merges templates into home dirs
• JSON configs: use jq -s '.[0] * .[1]' to merge (local overrides preserved)
• YAML configs (e.g. Hermes): no reliable shell-native merge — use create-or-skip strategy (copy template if target absent, skip if differs, --force to overwrite). This is safe when secrets live in separate files (auth.json, .env).
• TOML configs: use awk to strip managed keys, prepend defaults, append managed sections

What IS transferable across machines (sync via repo):

• Model defaults, toolset configs, terminal settings
• Skills (bundled update handles these; custom skills version in repo)
• Agent persona files (SOUL.md, AGENTS.md)
• Plugin/extension lists

What IS transferable via git snapshots (nightly cron → commit):

• Agent memories (Hermes MEMORY.md/USER.md, Codex project memory feedback_/project_ files)
• Learning artifacts (corrections/, patterns/, learned-patterns.json, cc-insights/)
• Cross-agent state (hermes-insights.yaml, cross-agent-memory.yaml)
• Codex learned rules (default.rules), history, session index
• Gemini state.json, projects.json
• Exported session JSONL (tool-call metadata, not full transcripts)

Pattern: snapshot to config/agents/{agent}/state-snapshots/ or config/agents/{agent}/memories/, commit nightly via scripts/cron/commit-learning-artifacts.sh, restore on new machine via sync-agent-configs.sh (copy snapshot → home dir if local files absent).

CRITICAL: .gitignore exceptions are useless without git add in the pipeline. Audit that every !.Codex/state/foo/ exception actually has files committed — discovered 5/10 exceptions had zero files in git index despite allowing tracking.

What is NOT transferable (machine-local, manual setup):

• Auth tokens, API keys (auth.json, .env, oauth_creds.json)
• Full session transcripts (too large for git — use rsync backup to secondary machine)
• SQLite databases (state.db, state_5.sqlite — too large, use rsync)
• File snapshots, checkpoints
• Model availability caches

Extending to new agents:

When adding a new agent (e.g. Hermes) to the sync pipeline:

1. Create config/agents/{agent}/config.yaml.template (strip secrets)
2. Add sync function to sync-agent-configs.sh
3. Add agent to harness-update cron schedule for all target machines
4. Verify with --dry-run before live sync

Post-OS-Update Machine Recovery

OS updates (apt upgrade, kernel updates) commonly break multi-machine readiness:

Display/VNC failures after update:

• /etc/gdm3/custom.conf may re-enable Wayland (resets WaylandEnable=false)
• Kernel updates change DRM card numbering (card2-DP-3 → card0-DP-1)
• NVIDIA driver updates change DFP-N output mappings
• Captured EDID files (/etc/X11/edid.bin) become stale
• KVM switches lose EDID signal → display goes blank

NVIDIA kernel module gap (most common post-update failure):

When nvidia-smi fails after a kernel upgrade, check this first:

ssh target "uname -r"                                    # running kernel
ssh target "dpkg -l | grep linux-modules-nvidia | grep -v '^rc'"  # installed nvidia module packages
# Compare: if running kernel (e.g. 6.17.0-20) has no matching nvidia module package → that's the problem
# Fix: sudo apt update && sudo apt install linux-modules-nvidia-580-open-$(uname -r)
# If package doesn't exist yet: boot into older kernel via grub-reboot
ssh target "ls /boot/vmlinuz-*"   # available kernels

The display falls back to simple-framebuffer (card0) when NVIDIA can't load — all DP outputs disappear.

Recovery checklist (run from primary via SSH):

1. ssh target "grep WaylandEnable /etc/gdm3/custom.conf" — verify X11 mode
2. ssh target "nvidia-smi" — verify driver loaded (if fails, check kernel module gap above)
3. ssh target "ls /sys/class/drm/ | grep DP" — check card numbering
4. Re-run display fix script if card numbering changed
5. Re-capture EDID if monitor connection changed
6. Test VNC connection after display is confirmed

Guard against future updates:

• Pin critical Xorg configs with dpkg-divert or apt conf.d hold
• Add post-update readiness check to cron (verify Wayland disabled, EDID present)
• Document hardware-level permanent fix (EDID emulator dongle) as alternative

Post-Update Drift Detection and Health Verification

The harness-update.sh daily cron updates tools but has critical blind spots. Address these when extending or auditing the harness:

Gap: No post-update smoke test

After updating a tool, always verify it works. --version is the minimum. Real-world failure: hermes update stashed/popped local changes, rewrote the launcher shebang to a hardcoded venv path. Reverting to the "portable" shebang (#!/usr/bin/env python3) broke Hermes entirely because system python3 lacks venv dependencies (ModuleNotFoundError: No module named 'dotenv').

Gap: No pre-update security/compatibility gate

Updates are applied blindly — no npm audit, no changelog review, no major-version-bump guard. A malicious or breaking npm update propagates to all machines within 24 hours.

Gap: harness-update.sh coverage is incomplete

As of 2026-04, it covers 4 tools (GStack, Hermes, Superpowers, GSD) but misses the 3 primary AI CLIs (Codex, Codex, Gemini CLI) which are npm globals.

Machine-specific vs. portable changes in git-based tools

Some files MUST differ per machine (e.g., Hermes hermes launcher with venv shebang). Others are portable (e.g., package-lock.json). When checking for drift:

# Drift classification pattern:
hermes:
  machine_specific:    # NEVER revert or sync — intentionally different per machine
    - "hermes"         # shebang must point to local .venv/bin/python3
  portable:            # SHOULD be identical across machines
    - "package-lock.json"
  never_sync:          # secrets, accumulated state
    - "auth.json"
    - ".env"
    - "memories/"

Key lesson: Don't blindly git checkout files in git-based tools after update. Classify first, revert only machine-specific artifacts that were accidentally made portable, and flag portable changes for commit to a fork.

Recommended harness-update extensions (tracked in issue #1668)

1. Post-update: tool --version smoke test; mark BROKEN + rollback on failure
2. Pre-update: block major version bumps; flag npm audit critical/high
3. Post-update: git status --porcelain on git-based tools; log dirty files
4. Coverage: add Codex, Codex, Gemini CLI update functions
5. Rollback: npm install -g pkg@previous for npm; git stash for git tools

Agent Learnings Portability Audit

When assessing whether learnings survive machine loss, run a 3-parallel-subagent audit:

1. Subagent A: Home directory inventory — scan ~/.Codex/, ~/.hermes/, ~/.codex/, ~/.gemini/ for all state files. For each: size, git-tracked?, export pipeline?, risk if lost. Focus on files <1MB that contain unique learnings (corrections, rules, patterns, memories) vs large regenerable caches.

2. Subagent B: Project memory audit — scan ~/.Codex/projects/*/memory/ for all per-project memory files. These are the highest-value-per-byte files (user corrections in feedback_*.md). Check if backup/rsync covers them vs git tracking.

3. Subagent C: Nightly pipeline artifact audit — trace every script in scripts/cron/ that produces output. For each output file: where written, gitignore status, actually committed?, survives clone? The most common failure: .gitignore has ! exception but pipeline never runs git add.

The gitignore-exception gap pattern

This is the #1 discovery from the audit. .gitignore allows:

!.Codex/state/corrections/
!.Codex/state/patterns/

But if the nightly pipeline never runs git add .Codex/state/corrections/, those files accumulate locally and are NEVER committed. The exception is dead code. Always verify exceptions have matching files in git ls-files.

Three-tier portability model

• Tier 1 (git-committed): Small files <10MB. Memories, patterns, corrections, rules, insights. Survives git clone.
• Tier 2 (rsync backup): Large files 10MB-1GB. Session transcripts, SQLite databases. Survives single-machine failure.
• Tier 3 (regenerable): Caches, plugins, debug logs. No backup needed.

Legal gate

ALL agent memory snapshots must pass legal-sanity-scan.sh before commit. Memory files and correction records routinely contain file paths, commands, or project names that reference client work.

Pitfalls

• Don't optimize repo-local issues before hub blockers that affect every session
• Don't install every provider on every machine without a role-based reason
• Don't treat umbrella issues as execution-ready without decomposition
• Don't confuse "reachable" with "ready"
• Don't burn premium provider quota on bounded implementation tasks
• Don't copy auth/secrets via repo sync — always manual per-machine setup
• Don't assume display configs survive OS updates — always re-verify after apt upgrade
• Don't revert shebang lines in git-based tools without checking if the venv path is intentional — system python3 likely lacks tool dependencies
• Don't assume harness-update.sh exit 0 means all tools work — it always exits 0 even when tools are broken