Plabs Lifecycle

plabs Lifecycle

Wrap plabs enable / plabs disable / plabs apply behind a shared file lock so multiple concurrent agents (batch module creation, workflows) can safely serialize their terraform applies. The lock prevents lost work — terraform's own state lock would just fail loudly on contention.

The implementation lives in scripts/plabs-lifecycle.sh. This skill teaches when and how to invoke it.

Why this exists

pathfinding-labs uses a single shared Terraform state at ../pathfinding-labs/terraform.tfstate. Only one plabs apply can run at a time. plabs enable X and plabs disable X just toggle scenario inclusion in the next apply — they're cheap on their own; the expensive step is the apply itself.

The natural batching unit is therefore the current set of enabled scenarios, not "5 modules." Adding one scenario to an already-deployed set is a small terraform delta. That's what makes the rolling-pool coordination model efficient.

Commands

Run from the pathrunner repo root:

./scripts/plabs-lifecycle.sh enable  <scenario-id>...           # enable + apply + import creds, holds lock
./scripts/plabs-lifecycle.sh disable <scenario-id>...           # disable + apply + clear identity, holds lock
./scripts/plabs-lifecycle.sh swap [--add <id>]... [--remove <id>]...  # combined, one apply, +import/-clear
./scripts/plabs-lifecycle.sh status                             # read-only, no lock
./scripts/plabs-lifecycle.sh lock-info                          # who holds the lock, how long
./scripts/plabs-lifecycle.sh force-unlock                       # emergency only

Companion scripts (invoked automatically by the lifecycle script, but usable standalone):

• ./scripts/check-sso.sh — verify SSO profile freshness (plabs profiles + optional pathrunner attacker profile)
• ./scripts/import-lab-creds.sh <scenario-id> — pull scenario starting creds via plabs credentials and add to pathrunner identity store

Scenario IDs are the plabs form (e.g., lambda-005-to-admin, iam-013-to-admin) — the pathfinding-cloud path ID plus the -to-{target} suffix. Confirm the exact ID via ./scripts/plabs-lifecycle.sh status or (cd ../pathfinding-labs && ./plabs scenarios list).

Prefer swap when both adding and removing in the same coordinator turn — it does one terraform apply instead of two.

Automatic side effects

Every enable, disable, and swap invocation does more than toggle terraform state:

1. Preflight SSO check — before acquiring the lock, check-sso.sh plabs probes every AWS profile plabs is configured to use. If any is expired, the script exits code 4 with the failing profile name and instructions (aws sso login --profile <name>). This avoids the failure mode where plabs apply runs for two minutes, then dies mid-terraform because a token silently expired. Bypass with SKIP_SSO_CHECK=1 if you need to (e.g., testing offline).
2. Auto-import scenario credentials — after a successful enable or swap --add, import-lab-creds.sh fetches each newly-enabled scenario's starting IAM user credentials via plabs credentials <id> --format=json and registers them in pathrunner's identity store under the name <scenario-id>. Idempotent: if the identity already exists (e.g., previous run), it's replaced with the current keys. Bypass with SKIP_CREDS_IMPORT=1.
3. Identity cleanup on disable — after a successful disable or swap --remove, each removed scenario's pathrunner identity (identity clear <scenario-id>) is removed. Prevents stale identities pointing at IAM users terraform just destroyed.

The SSO check is fast (~1s per profile), so it's fine to run on every invocation. The auto-import is only invoked on paths that succeed — if plabs apply fails, no creds get imported.

Downstream sub-agents doing module build/test work don't need to run plabs credentials themselves; they can just ./pathrunner identity switch <scenario-id> after enable finishes.

The lock

• Location: /tmp/pathrunner-plabs.lock.d (override with PLABS_LOCK_DIR).
• Implementation: atomic mkdir — portable across macOS/Linux, no flock dependency.
• Owner file: records PID, acquisition timestamp, hostname, purpose.
• Wait timeout: default 3600s (PLABS_LOCK_TIMEOUT). Exit code 2 on timeout.
• Stale reclamation: if the lock is older than PLABS_LOCK_LEASE_MAX (default 1800s) AND the recorded PID is no longer alive on this host, the next acquire reclaims it automatically. This handles agents that were SIGKILLed mid-apply.
• Read-only commands (status, lock-info) do NOT acquire the lock.

If lock-info shows a stale lock and reclamation didn't fire (rare — e.g., PID reused, or a run <30 min but the shell is gone), use force-unlock after verifying no plabs apply is actually running elsewhere: pgrep -af 'plabs apply'.

When to invoke each command

• enable — one-off, when you're starting work on a single module. Use swap instead if you're also removing something.
• disable — one-off, when you're done with a scenario and no new one is starting in its place.
• swap — the coordinator's workhorse. When one module finishes and another starts, swap --remove <finished> --add <next> in a single call does both toggles + one apply.
• status — before starting work, to see what's already deployed and skip redundant enables.
• lock-info — when a command is waiting longer than expected, to see who's holding things up.

Integration with module creation

The recommended coordination model (chosen ADR-style with the user) is a rolling pool of 5 deployed scenarios:

1. Coordinator picks the first 5 gaps from /list-gaps output and calls plabs-lifecycle.sh enable id1 id2 id3 id4 id5 once. One apply deploys all five.
2. Five sub-agents work in parallel — each on one module. They can plabs scenarios credentials <id> to get the starting user, and run tests against the deployed lab. The build/test phase itself does NOT touch this skill or the lock — the lab is already deployed and their work is read-only against AWS.
3. When any sub-agent finishes (module + tests passing), the coordinator picks the next unstarted gap and calls plabs-lifecycle.sh swap --remove <finished-id> --add <next-id> — one apply retires the old, deploys the new.
4. Repeat until the gap list is drained; final plabs-lifecycle.sh disable <last-N> tears down the remaining pool.

The lock keeps steps 1, 3, and 4 safe under concurrency without requiring the sub-agents to coordinate directly. A future batch-modules skill or workflow will implement this loop; this skill provides the primitives.

Failure modes and recovery

• plabs apply fails mid-run: terraform state on disk is authoritative. Re-run the same enable/disable/swap command — it's idempotent because terraform will replay the intended state.
• Lock timeout (exit 2): check lock-info. If a real apply is in progress, wait; if it looks stuck, investigate before force-unlocking.
• Scenario ID not found: verify the exact ID with (cd ../pathfinding-labs && ./plabs scenarios list) | grep <partial>.
• plabs status shows a scenario as "enabled but not deployed": a previous apply failed. Just re-run enable for that ID (or swap --add for it) to trigger a fresh apply.
• Attacker infra collision: if a pathrunner workflow also deploys attacker infra ec2 create at the same time as a lab apply, they're independent terraform states (attacker infra is in pathrunner's tracker, lab infra is in pathfinding-labs). No lock coordination between them is needed.

What NOT to do

• Don't run plabs apply directly in the labs repo while any agent might be using this script — bypasses the lock.
• Don't set PLABS_LOCK_TIMEOUT=0 to fail-fast unless you're intentionally probing lock state — a real apply takes minutes.
• Don't force-unlock without checking pgrep -af 'plabs apply' and lock-info first — you can corrupt a running apply.
• Don't hold the lock across the module build/test phase — enable, release (implicit on script exit), work, then re-acquire to disable. The script already does this correctly per invocation.