Live Aware Account Routing

Live-Aware SLURM Account Routing (Apr 8, 2026)

Experiment Overview

Context — what failed before this fix

KINTSUGI's _build_cycle_assignment() in workflow/Snakefile pre-assigned cycles to accounts at DAG creation, reading static gpu_slots from each project's workflow/config.yaml. The _qc_cpu_assignment() and _registration_assignment() helpers always picked the first account in config order. None of them queried live SLURM availability, even though the underlying detector (detect_live_multi_account() in src/kintsugi/hpc.py:687) already counted ALL users on each account's QOS pool via slurmInfo / squeue.

Symptom: 5 SLURM jobs from project src_CX_19-002_spleen_CC2-A_D200210 stuck PENDING (QOSGrpMemLimit) against the clive account because user drew.bloss was running a 256 GB job in the same investment QOS pool (limit: 312.5 GB total). KINTSUGI's 4 decon jobs (125 GB each) and 1 qc_stitch (62.5 GB) couldn't fit in the 56.5 GB of remaining headroom.

Verification of static-assignment determinism: With clive.gpu_slots=1, maigan.gpu_slots=2, round-robin pre-assignment for 9 cycles always pinned cyc01, 05, 07, 09 → clive. The 4 stuck decon jobs matched exactly. The qc_stitch hit clive because clive was first in the config.

Root cause — feature existed but wasn't wired up

detect_live_multi_account() returns per-account gpu_avail, cpu_avail, usage (across all users). The CLI called it (cli.py:2041) but only consumed pool["total_gpu_avail"] to set -j. Per-account live data was thrown away. The Snakefile assignment helpers never saw it.

Verified workflow

Step 1 — Inject live data from CLI to Snakemake

src/kintsugi/cli.py _workflow_run() builds a live_accounts list from pool["accounts"] and forwards via --config:

live_accounts = [
    {
        "name": a["name"],
        "gpu_avail": a.get("gpu_avail", 0),
        "cpu_avail": a.get("cpu_avail", 0),
        "mem_avail_gb": max(0, a["alloc"]["mem_gb"] - a["usage"]["mem_gb"]),
    }
    for a in pool["accounts"]
]
if pool["total_gpu_avail"] == 0 and pool.get("total_cpu_avail", 0) == 0:
    raise SystemExit(1)  # Hard-fail instead of queueing forever
cmd.extend(["--config", f"live_accounts={json.dumps(live_accounts)}"])

The actual snakemake invocation includes:

--config live_accounts=[{"name": "maigan", "gpu_avail": 2, "cpu_avail": 4, "mem_avail_gb": 593}]

Step 2 — Snakefile reads live data

workflow/Snakefile:

_live_raw = config.get("live_accounts", None)
if isinstance(_live_raw, str):
    import json as _json
    LIVE_ACCOUNTS = _json.loads(_live_raw)
elif isinstance(_live_raw, list):
    LIVE_ACCOUNTS = _live_raw
else:
    LIVE_ACCOUNTS = None

def _live_lookup(name):
    if not LIVE_ACCOUNTS:
        return None
    for la in LIVE_ACCOUNTS:
        if la.get("name") == name:
            return la
    return None

def _effective_gpu_slots(acct):
    la = _live_lookup(acct["name"])
    if la is not None:
        return max(0, int(la.get("gpu_avail", 0)))
    return acct.get("gpu_slots", 0)

_build_cycle_assignment() swaps acct.get("gpu_slots", 0) for _effective_gpu_slots(acct). Saturated accounts contribute 0 slots and are skipped entirely. Static fallback path is preserved for bare snakemake invocations.

Step 3 — Live-aware QC and registration

_qc_cpu_assignment() sorts CPU candidates by (mem_avail_gb, cpu_avail) descending and picks the top one if it has nonzero memory headroom. Falls back to first-account if LIVE_ACCOUNTS is None. _registration_assignment() sorts GPU candidates by (gpu_avail, mem_avail_gb) similarly.

Step 4 — Permanent block of saturated account

When an account is chronically saturated by other users (clive's QOS was throttled to 1 GPU/312.5 GB and is regularly hammered), add it to BLOCKED_ACCOUNTS in src/kintsugi/hpc.py:

BLOCKED_ACCOUNTS: frozenset[str] = frozenset({"brusko", "clive"})

Future kintsugi workflow config invocations will skip blocked accounts entirely.

Step 5 — Bulk-strip blocked accounts from existing project configs

kintsugi workflow config only writes new configs — existing workflow/config.yaml files retain stale account lists. Use a Python script to bulk-update:

import yaml
from pathlib import Path
for cfg in Path("/blue/maigan/smith6jt/KINTSUGI_Projects").glob("*/workflow/config.yaml"):
    data = yaml.safe_load(cfg.read_text())
    accts = data["resources"]["accounts"]
    if not any(a["name"] == "clive" for a in accts):
        continue
    new = [a for a in accts if a["name"] != "clive"]
    data["resources"]["accounts"] = new
    data["resources"]["total_gpu_slots"] = sum(a.get("gpu_slots", 0) for a in new)
    data["resources"]["total_cpu_slots"] = sum(a.get("cpu_slots", 0) for a in new)
    data["resources"]["total_slots"] = data["resources"]["total_gpu_slots"] + data["resources"]["total_cpu_slots"]
    cfg.write_text(yaml.safe_dump(data, default_flow_style=False, sort_keys=False))

Verified Apr 8 2026: 34/35 project configs updated, 0 remaining clive references.

Failed attempts (Critical)

Recognizing the failure mode

Final configuration

cli.py injection point (~line 2041 in _workflow_run)

After the existing pool = detect_live_multi_account(...) call, add the live_accounts list build, hard-fail on zero capacity, and --config live_accounts=<json> extension to cmd.

Snakefile parser block (top of multi-account section, ~line 124)

LIVE_ACCOUNTS, _live_lookup(), _effective_gpu_slots() — see Step 2 above.

Snakefile assignment helpers (~lines 130–230, 369–460)

• _build_cycle_assignment(): use _effective_gpu_slots(acct) everywhere instead of acct.get("gpu_slots", 0). Add fallback path that re-uses static gpu_slots if LIVE_ACCOUNTS zeroed everything (so bare snakemake runs still work).
• _registration_assignment(): sort by live (gpu_avail, mem_avail_gb) and pick top when nonzero; fall back to first GPU account otherwise.
• _qc_cpu_assignment(): sort by live (mem_avail_gb, cpu_avail) and pick top when nonzero; fall back to first CPU-capable account.

hpc.py BLOCKED_ACCOUNTS

Document why each account is blocked in a comment block above the frozenset. The "why" is what lets future maintainers re-evaluate.

Verification checklist

1. CLI smoke test: kintsugi workflow run . --dry-run should show --config live_accounts=... in the printed command.
2. Snakefile smoke test: snakemake --config 'live_accounts=[{"name":"clive","gpu_avail":0,"cpu_avail":0,"mem_avail_gb":0},{"name":"maigan","gpu_avail":2,"cpu_avail":8,"mem_avail_gb":625}]' -np stitch should route every cycle to maigan.
3. Live verification: After launching kintsugi workflow batch -d <dataset> --detach, run scontrol show job <jobid> | grep -E "Account|Comment" and confirm Account=maigan for every submitted job.
4. Regression: With both accounts healthy, cycle distribution should still follow proportional gpu_slots (preserved in static fallback path).

Key insights

• Detection without consumption is worse than no detection — KINTSUGI had detect_live_multi_account() for months before this fix; the data was just thrown away.
• Static config drifts faster than you think — clive's QOS was silently throttled from 3 GPU/812 GB → 1 GPU/312.5 GB and the config files weren't reconciled. Always verify against live QOS limits, not docs.
• First-account bias is a hidden failure mode — every "pick the first account" loop is a single point of failure when that account misbehaves. Sort by live availability instead.
• Snakemake --config supports JSON — use json.dumps() for nested structures, not Python repr.
• Hard-fail beats infinite queue — if all accounts are saturated, abort with an actionable error message. Let the human investigate.
• Permanent blocks need rationale comments — BLOCKED_ACCOUNTS = frozenset({"brusko"}) is opaque. Document why so future maintainers can re-evaluate.
• Bulk config updates need yaml-aware tools — sed is fragile for nested YAML. Use yaml.safe_load/safe_dump in a small Python script.

When to apply this pattern

• Multi-tenant SLURM with shared investment QOS pools
• Account-level resource limits that include OTHER users' jobs in the count
• Snakemake DAG creation that pre-assigns jobs to accounts
• Static config files that drift from live cluster state
• Any "pick the first thing in the list" routing logic on multi-account systems

References

• Plan file: /home/smith6jt/.claude/plans/velvety-juggling-thimble.md
• KINTSUGI workflow CLAUDE.md — "Multi-Account Architecture" section
• slurm-concurrent-processing skill — original multi-account architecture (now superseded by live-aware variant)
• src/kintsugi/hpc.py:370 — detect_current_usage() (counts ALL users on a QOS, key for noisy-neighbor detection)
• src/kintsugi/hpc.py:687 — detect_live_multi_account() (per-account live availability)
• HiPerGator account/QOS limits: https://help.rc.ufl.edu/doc/Account_and_QOS_Limits