Capacity Pool

Shared PLI capacity pool

Deployment scope

This is a published snapshot of dangxingyu's existing PLI/Della deployment, not a standalone scheduler installer. The absolute paths, account, partitions, slot names and runtime policy below belong to that deployment. Before using it elsewhere, obtain the site's authorized pool CLI and deployment configuration; do not create these paths, submit jobs under another user's account, or infer access from installing this skill. If the documented CLI or deployment is absent, report the missing prerequisite and stop before any scheduler mutation.

On the existing Della installation, the project-local skill remains the operational source; publishing this snapshot does not replace its discovery links, ledger, controllers, immutable releases or queued jobs.

Canonical deployment

One implementation: /scratch/gpfs/ARORA/xd7812/wuji/projects/capacity-pool. One CLI: /home/xd7812/.local/bin/pool. One canonical ledger: /scratch/gpfs/ARORA/xd7812/capacity-pool. Codex and Claude use this same skill, not project-specific copies. Its only source is that project's skills/capacity-pool/; .agents, .codex and .claude skill discovery paths link to it. Edit the project source. The project README is the navigation entrypoint for code, live state, frozen releases and archives. They are different roles, not separate pool managers. Do not create another capacity-pool project or copy this skill per experiment. The old pli-gpu-capacity discovery name is a redirect whose tools also live in this project. Retired legacy data is in archive/legacy-reservations; ~/pli-gpu-pool retains only data aliases, its original lock and a retirement marker. Legacy writes (including status --sync) are closed; plain legacy status is historical, not current capacity or another submission entrypoint. For a request to leave only one physical directory, source/skill consolidation and navigation links are not completion. Report the external state/release paths still present and use the operator's coordinated cutover gates.

Start with pool status --json. It joins live Slurm state, native agents, and registered legacy holders during migration. Empty native observations do not mean free GPUs. allocated_gpus excludes pending jobs; scheduler start times are estimates. Read the errors and migration state, not just job existence.

Choose the role from the user's request

• Status, explanation, diagnosis: read-only inspection. Do not silently submit, cancel, reassign, or restart anything.
• Run experiments: consumer workflow below, only on the authorized slot.
• Maintain, deploy, unify, or repair capacity: read references/operator.md. The user's maintenance request authorizes necessary in-scope operator actions; do not confuse consumer restrictions with a prohibition on authorized maintenance. It does not authorize stopping another project's workload or increasing the requested capacity.

Use pli-cluster only for cluster access when necessary. On Della, run local commands directly; no relay is required.

Experiment consumer

pool request --id UNIQUE --owner OWNER --slot h100-32b \
  --project PROJECT --description 'PROJECT: one sentence describing the experiment.' \
  --manifest /absolute/manifest.json --run-root /absolute/run-root \
  --lane-role /absolute/lane-role.sh --lease-minutes 480
pool renew UNIQUE --minutes 480
pool release UNIQUE
pool stop h100-32b --request-id UNIQUE
pool cancel UNIQUE

--slot is mandatory. Repeat only if the user authorized either slot. A and B are separate 4-node × 8-GPU allocations, not one 64-GPU DDP allocation. A manifest and lane role describe work; the slot agent launches its one owned outer Slurm step. Consumers never attach with personal srun, cancel holder jobs/steps, start watchers, or hand-edit ledger files. Renew active leases; expiry/release drains owned work and restores warm. A pool request does not convert a legacy holder to native or take over its old ACTIVE reservation. An operator-registered legacy-pool backend can serve the same request API during migration; inspect its certificate and exact work-step binding. It is not a native holder. Do not start its broker as an experiment consumer.

Invariants for both roles

• New jobs, including tests: Della account pli, partition pli-c, QOS pli-short only. No arora / gpu-test fallback. Check actual resources, account, QoS, walltime and the current user limit before submission.
• Production A/B holders and their NEXT must request the PLI-short ceiling: 24 hours (1-00:00:00), not the duration of the experiment or a legacy 4-hour predecessor. Short isolated tests require explicit test scope and no auto-successor. Verify the actual Slurm TimeLimit after submission/update; a script header or an old SubmitLine is not the effective job setting. When authorized to repair an already queued short holder, use the operator procedure to preserve its job ID/queue age; do not cancel and resubmit.
• Capacity renewal is independent of experiments and who owns their work. Both A and B need CURRENT plus an already-submitted NEXT while CURRENT runs. Admitted native agents queue NEXT on their first healthy cycle (24-hour lead). Runtimes with sbatch/renewal-policy.json mode independent-standby submit without afterany or singleton. Legacy frozen runtimes retain both; never remove dependencies from an old-runtime job in place. Independent submissions verify Slurm enforces the existing pli-short 64-GPU/user ceiling. An early NEXT keeps its own GPUs warm, without claiming the slot lease or submitting another NEXT. Only terminal predecessor ancestry plus the slot lock permits admission and work. This version does not reassign A/B leases. Missing NEXT is an alarm. Queuing is not a gap-free allocation guarantee. Della has no ACCRUE_ALWAYS: dependency waiting does not accrue age. Inspect effective AccrueTime and sprio AGE; do not substitute SubmitTime or assume age accrues under every other scheduler limit without observing it. A short holder reaching TIMEOUT is a lost allocation, even if NEXT exists; report the gap. Pending capacity cannot be kept warm before allocation.
• Idle allocated GPUs stay warm. A live process or requested warm mode is not evidence of activity: check fresh job/step/node bindings and all physical GPU receipts/probes. Failed observations mean UNKNOWN, never empty/free. The read-only dashboard is pool dashboard (loopback port 8794). It shows shared queue, leases, CURRENT/NEXT and per-GPU observations. idle means a sampled utilization of at most 1%, not permission to take a leased GPU.
• One GPU controller per allocation. Never run a new controller beside an old one. Native control lives inside the batch allocation; the login migration bridge is temporary and does not launch work or GPU steps. For a certified legacy work transport, the existing watcher remains the sole fill/idle controller; the pool broker is only its journaled work launcher. Do not add a native controller or a personal launcher to that allocation.
• The designated login host's user systemd units serve the read-only dashboard and run a bounded recovery check each minute/after boot. Recovery is part of this same project/CLI, not a second GPU controller: it never runs srun or changes leases. Ordinary recovery only replaces an entirely terminal chain. Separately, an explicitly authorized pool admin stage-upgrade plan can replace only the pending NEXT of its exact CURRENT after CURRENT starts; it preserves CURRENT and uses the journaled replacement API. It never invents upgrades from ordinary recovery authority or retargets another generation. Both paths require exact identity, locks and submission-intent checks. A failed observation fences submissions. See the operator reference before managing these units. Enabled is not the same as started: check conditions, active timer, fresh recovery receipt and dashboard HTTP after maintenance. The deployed login units use the designated machine's ID as the boot host fence, not its early hostname; a host reimage requires authorized re-rendering.
• Exact identities and locks govern changes. No broad scancel, pkill, or job-name guesses. A lost submission acknowledgement requires reconciliation of its durable intent/token, never a blind retry.
• Preserve running work and retained queued legacy jobs. Their spooled scripts do not change when repository files change. CLI installation, native GPU activity, and end-to-end rollover acceptance are separate milestones.

For status, report allocated/work/warm/standby/pending separately, with each slot's actual holder time limit, NEXT time limit and lease owner. Once native rollout has begun, follow canonical successor journals beyond the initial migration generation. A stale legacy watcher must not hide a later pending native job. Neither may a terminal native heartbeat. Follow the confirmed successor chain; do not reuse predecessor node/work receipts for a new job. A canonical lease still denotes ownership even when its allocation is pending or unobserved. Do not report full deployment acceptance from unit tests or a corrected CLI alone: verify the actual running/queued releases and their effective policy.

Deployment details and test evidence: projects/capacity-pool/README.md. pli-gpu-capacity is a compatibility redirect, not a second operating system.

Scheduler-facing job names

User preference: use concise, truthful purpose names such as training, meta-grad-training, or optimizer-training. Do not put A/B slot suffixes, GPU counts, or hardware labels in public job names; this supersedes the earlier wuji-research-a/b suggestion. Use self-play only for actual self-play work. Keep resource requests/accounting and work/warm status accurate. Slot chains retain distinct purpose names. Check collisions, including old runtimes whose Slurm singleton dependencies are keyed by user and name.

Keep h100-32a / h100-32b as stable INTERNAL slot/ledger/CLI identifiers. The target frozen runtime's sbatch/job-names.json governs public names. Purpose-name runtimes specify optimizer-training / meta-grad-training; earlier named runtimes specify wuji-research-a/b, and old runtimes without a contract require JobName == slot. The CLI validates the target runtime's contract, not arbitrary aliases. Submitters must export SLOT_NAME explicitly. New native NEXT journals record runtime and public name with the token.

Do not rename a queued/running job with standalone scontrol update JobName=.... Installing a new CLI does not upgrade a spooled job's runtime. For explicitly authorized NEXT replacement (new job ID/queue age), use pool admin replace-next and the operator reference. It preserves CURRENT and the cancelled NEXT's original journal, then appends a new-runtime successor. The target runtime determines whether Slurm dependencies or allocation-local standby protect the handoff. MIGRATING means the replacement is registered, not that work has rolled over. RUNNING standby counts as allocated GPUs but never as a lease or active experiment; PENDING counts zero. Never erase the cancellation tombstone or restart CURRENT to clear its expected old-NEXT warning. Preserve leases and verify the actual replacement and its subsequent successor names.

Experiment descriptions

Every new CLI request requires --project and --description: one line of 1–240 characters naming the project and what the experiment tests. Write a useful scientific description in English, not just a run ID. The dashboard interface may use Chinese; experiment descriptions default to English. Do not include secrets. Examples: Kron-ISO: compare two inner learning rates on the full-F/C objective. or Infinite-MG: rerun frozen centre recipes on H100 for matched-hardware comparisons.

Descriptions are display metadata, not scheduling instructions. Immutable metadata/<request-id>.json records preserve them across old frozen controllers which do not know the new field. Legacy records remain readable; an authorized pool describe ID --project PROJECT --description '...' adds a missing summary without rewriting a queue entry, lease, manifest or result. Do not backfill scientific claims from a run name alone. The dashboard's DONE/FAILED counts are execution markers, not scientific acceptance or optimizer rankings.