# Palmetto Apptainer Libcuda Fix

> Fix Palmetto Slurm jobs where Apptainer or vLLM workers fail with `libcuda.so cannot found` even though `/lib64/libcuda.so.1` exists on the host. Use when logs show CUDA driver discovery failures inside containerized GPU jobs and you need a safe repair plus smoke validation before resubmitting the full run.

- Skill: `kwongfuk/palmetto-apptainer-libcuda-fix` (Agent Skill)
- Install (CLI): `npx skillmds@latest add kwongfuk/palmetto-apptainer-libcuda-fix`
- Raw SKILL.md: https://api.skillmd.com/api/skills/kwongfuk/palmetto-apptainer-libcuda-fix/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Product & Planning
- Author: KwongFuk (https://skillmd.com/u/kwongfuk)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/kwongfuk/palmetto-apptainer-libcuda-fix

---


# Palmetto Apptainer libcuda Fix

Use this skill when a Palmetto GPU job fails inside Apptainer with messages like:
- `libcuda.so cannot found`
- `Possible files are located at ['/lib64/libcuda.so.1']`
- vLLM engine core startup fails before the model becomes ready

## Goal

Repair container-side CUDA driver discovery without hand-waving. Do not just append more paths to `LD_LIBRARY_PATH` and hope. Make the missing soname visible at the exact path the containerized workers need, then prove it with a short smoke run before resubmitting the real job.

## Required workflow

1. Confirm the failure signature.
- Read the Slurm stdout/stderr or service log.
- Verify the root cause is specifically missing `libcuda.so` inside the containerized worker path.
- If the failure is a different CUDA error, do not use this fix blindly.

2. Patch the sbatch launcher.
- Add a scratch-local compatibility directory, for example:
  - `CUDA_COMPAT_DIR="$JOB_TMP_ROOT/cuda_compat"`
  - `CUDA_COMPAT_LIB="$CUDA_COMPAT_DIR/libcuda.so"`
- Create the symlink on the host:
  - `mkdir -p "$CUDA_COMPAT_DIR"`
  - `ln -sfn /lib64/libcuda.so.1 "$CUDA_COMPAT_LIB"`
- Bind that exact file into the container at the missing soname path:
  - `CUDA_COMPAT_BIND="$CUDA_COMPAT_LIB:/lib64/libcuda.so"`
  - `APPTAINER_GPU_BINDPATH="${APPTAINER_BINDPATH},${CUDA_COMPAT_BIND}"`
- For the GPU-facing `apptainer exec --nv --cleanenv` calls, use:
  - `--bind "$APPTAINER_GPU_BINDPATH"`
  - `--env LD_LIBRARY_PATH="/lib64:$SCRATCH_ROOT/envs/vllm_env/lib"`
- Keep non-GPU preflight calls on the normal bind path unless they also need the driver.

3. Add a real preflight before the main server starts.
- Run a short `apptainer exec --nv --cleanenv` Python check that does all of:
  - `ctypes.CDLL("libcuda.so")`
  - `ctypes.CDLL("libcuda.so.1")`
  - optional `torch.cuda.is_available()`
- Fail fast here instead of waiting for a long vLLM startup timeout.

4. Run a smoke validation job.
- Submit a short one-GPU Slurm job that uses the same bind strategy as the real launcher.
- The smoke must confirm both parent and child Python processes can load:
  - `libcuda.so`
  - `libcuda.so.1`
- On Palmetto, prefer a short V100 or P100 smoke if H100 priority is slow. This checks driver visibility, which is the point of the fix.

5. Only then resubmit the real job.
- Reuse the patched launcher.
- Preserve any previous environment overrides such as `MODES`, `SKIP_EXISTING_SCENARIOS`, or custom output roots.
- Watch the new run until it clears the previous failure stage. Do not claim success just because the job is queued.

## Patch pattern

Use this pattern in the sbatch file:

```bash
CUDA_COMPAT_DIR="$JOB_TMP_ROOT/cuda_compat"
CUDA_COMPAT_LIB="$CUDA_COMPAT_DIR/libcuda.so"
CUDA_COMPAT_BIND="$CUDA_COMPAT_LIB:/lib64/libcuda.so"
APPTAINER_GPU_BINDPATH="${APPTAINER_BINDPATH},${CUDA_COMPAT_BIND}"

mkdir -p "$CUDA_COMPAT_DIR"
ln -sfn /lib64/libcuda.so.1 "$CUDA_COMPAT_LIB"

apptainer exec --nv --cleanenv \
  --bind "$APPTAINER_GPU_BINDPATH" \
  --env LD_LIBRARY_PATH="/lib64:$SCRATCH_ROOT/envs/vllm_env/lib" \
  "$APPTAINER_IMAGE" \
  "$VLLM_PYTHON" - <<'PY'
import ctypes
ctypes.CDLL("libcuda.so")
ctypes.CDLL("libcuda.so.1")
print("container_cuda_driver_ok")
PY
```

Use the same `APPTAINER_GPU_BINDPATH` and `LD_LIBRARY_PATH` in the real server launch.

## Acceptance criteria

Do not consider the fix complete until all are true:
- the smoke job exits `COMPLETED`
- smoke stdout shows both parent and child process driver loads succeeded
- the real benchmark job has been resubmitted
- the real job advances past the old `libcuda.so cannot found` stage

## Notes

- This is a container driver-visibility fix, not a general CUDA fix.
- The key move is binding a host-side symlink into `/lib64/libcuda.so` inside the container, because some worker paths look for that soname directly.
- If the workload later fails with OOM, model config, or networking errors, that is a new problem and should be debugged separately.