Nvidia Tensorrt LLM Exec Slurm Compile

Compile TensorRT-LLM on SLURM Cluster

Submit, monitor, and verify a TensorRT-LLM compilation job on a SLURM cluster using enroot containers.

When to Use

Finding the Docker Image

The official Docker image tag for a given TensorRT-LLM version is recorded in the repo itself:

<repo_dir>/jenkins/current_image_tags.properties

Read this file to find the current image URL (e.g., urm.nvidia.com/sw-tensorrt-docker/tensorrt-llm:pytorch-25.12-py3-aarch64-ubuntu24.04-trt10.14.1.48-skip-tritondevel-202602011118-10901).

Pre-dumping the Container Image (enroot import)

SLURM clusters using enroot/pyxis require a .sqsh container image. To avoid download overhead at compile time, pre-dump the image in advance using the enroot-import companion script:

# Basic usage — submits a SLURM job on a CPU partition to import the image
enroot-import --partition cpu_datamover --debug <docker_image_url>

The script submits an sbatch job that runs enroot import docker://<image_url> and produces a .sqsh file in the current directory. The output on stdout is the SLURM job ID.

enroot-import flags

enroot-import workflow

1. Read the image tag from jenkins/current_image_tags.properties in the TRT-LLM repo.
2. Run enroot-import to submit the import job:

   cd <directory_where_sqsh_should_be_stored>
   <path_to>/enroot-import --partition cpu_datamover --debug <image_url>
   

   IMPORTANT: Convert urm.nvidia.com/sw-tensorrt-docker/tensorrt-llm:xxx to urm.nvidia.com#sw-tensorrt-docker/tensorrt-llm:xxx to avoid credential issues.
3. Wait for the import job to complete (squeue -j <job_id>).
4. The resulting .sqsh file is the container_image used in the compile step.

Prerequisites

The user must provide (or you must ask for) these values:

Optional parameters:

Companion Scripts

This skill includes three companion scripts in scripts/:

Scripts directory: skills/nvidia-tensorrt-llm-exec-slurm-compile/scripts/

Instructions

Follow these steps in order:

Step 0: Resolve the Container Image (if needed)

If the user does not already have a .sqsh container image:

1. Read the Docker image tag from <repo_dir>/jenkins/current_image_tags.properties.
2. Use enroot-import to pre-dump it:

   cd <directory_for_sqsh_files>
   <scripts_dir>/enroot-import --partition cpu_datamover --debug <image_url>
   

3. Monitor the import job with squeue -j <job_id>.
4. Once complete, the .sqsh file path becomes the container_image parameter.

If the user already has a .sqsh file, skip this step.

Step 1: Gather Information

Ask the user for any missing prerequisite values listed above. At minimum you need:

• container_image (or the Docker image URL — then run Step 0 first)
• repo_dir
• mount_dir
• partition and account

If the user has used this workflow before, check if previous values are stored in memory files.

Step 2: Prepare the Scripts Directory

The compile scripts must be accessible from inside the container (i.e., under mount_dir). Either:

Option A — Copy companion scripts to a location under mount_dir:

scripts_dir=<mount_dir>/<username>/workspace/tensorrt_llm_scripts
mkdir -p ${scripts_dir}/log
cp skills/nvidia-tensorrt-llm-exec-slurm-compile/scripts/compile.sh ${scripts_dir}/
cp skills/nvidia-tensorrt-llm-exec-slurm-compile/scripts/compile.slurm ${scripts_dir}/
chmod +x ${scripts_dir}/compile.sh ${scripts_dir}/compile.slurm

Option B — If the user already has scripts at a known location, use those directly.

Step 3: Submit the Job

Run sbatch from the login node (or a node with SLURM client access):

sbatch \
    --nodes=1 --ntasks=1 --ntasks-per-node=1 \
    --gres=gpu:<gpu_count> \
    --partition=<partition> \
    --account=<account> \
    --job-name=<jobname> \
    --time=<time_limit> \
    <scripts_dir>/compile.slurm \
    <container_image> <mount_dir> <scripts_dir> <repo_dir>

Capture and report the job ID from the sbatch output.

Step 4: Monitor the Job (Proactive — Do NOT Wait for User)

You MUST actively poll the job until it completes. Do not submit and walk away.

# Check job status (repeat every 30-60 seconds)
squeue -j <job_id> -o "%.18i %.9P %.30j %.8u %.2t %.10M %.6D %R"

# Once running, periodically tail the log (do NOT use tail -f, use tail -30 instead)
tail -30 <scripts_dir>/log/compile_<job_id>.srun.log

Monitoring loop:

1. Poll squeue -j <job_id> to check state
2. If PD (pending) — report the reason, keep polling every 30-60s
3. If R (running) — tail the build log every 30-60s; look for [XX%] Building, errors, or completion
4. If the job disappears from squeue, it has finished — proceed to Step 5
5. If F (failed) — immediately read the full log and report the error

Progress indicators to look for in the log:

• [XX%] Building CXX object... — compilation progress
• Linking CXX... — link phase
• FAILED:, error:, fatal error: — build failure
• Successfully built — success

Step 5: Verify the Build

Once the job completes, check for success:

# Check SLURM exit code
sacct -j <job_id> --format=JobID,State,ExitCode,Elapsed

# Check the build log for errors
tail -50 <scripts_dir>/log/compile_<job_id>.srun.log

A successful build ends with a message like Successfully built tensorrt_llm or completes without error.

Common Build Flags Reference

Common architecture values:

• "100-real" — Blackwell (B200, GB200)
• "90-real" — Hopper (H100, H200)
• "89-real" — Ada Lovelace (L40S)
• "80-real" — Ampere (A100)
• "90;100-real" — Multiple architectures

Troubleshooting

Example Interaction

User: "Compile TRT-LLM on the OCI cluster"

Agent actions:

1. Ask for container image path, repo path, mount dir (if not known)
2. Confirm partition/account for OCI cluster
3. Copy scripts to accessible location under mount_dir
4. Submit with sbatch
5. Report job ID
6. Monitor with squeue until complete
7. Check logs and report success/failure