# K8S Launch Kit Config

> Use this skill when the user needs help understanding, creating, or editing a k8s-launch-kit (l8k) configuration file (l8k-config.yaml or cluster-config.yaml). Activate for: config file questions, parameter tuning, subnet configuration, NV-IPAM setup, DOCA driver settings, maintenance concurrency, NIC configuration operator settings, changing MTU, VFs, resource names, or understanding what any config field does.

- Skill: `nvidia/k8s-launch-kit-config` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add nvidia/k8s-launch-kit-config`
- Raw SKILL.md: https://api.skillmd.com/api/skills/nvidia/k8s-launch-kit-config/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: NVIDIA (https://skillmd.com/u/nvidia)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/nvidia/k8s-launch-kit-config

---


# l8k: Configuration Files

> **PREREQUISITE:** Read `../k8s-launch-kit-shared/SKILL.md` for install paths, global flags, and output modes.

Understand, create, or edit l8k configuration files.

## File Types

| File | Source | Purpose |
|------|--------|---------|
| `cluster-config.yaml` | Generated by `l8k discover` | Hardware inventory plus the resolved deployment profile |
| `l8k-config.yaml` | User-created or copied from cluster-config | Full config with both hardware + deployment settings |

## Usage

```bash
# Use a config file for generation
l8k generate --user-config my-config.yaml \
  --save-deployment-files ./output

# Combine user config with live discovery (hardware refreshed, settings kept)
l8k discover --user-config my-config.yaml \
  --kubeconfig ~/.kube/config \
  --save-cluster-config ./updated-config.yaml
```

## Profile Resolution and Write-Back

Fresh discovery and file-backed generation resolve and persist settings with
this precedence:

1. Hardware and built-in defaults fill missing fields.
2. Existing config values.
3. Explicit CLI flags override both.

Discovery with `--user-config` follows a stricter refresh contract: only
`clusterConfig` is replaced. Every other section remains as loaded unless an
explicit CLI flag overrides its corresponding field. Generation still fills
missing profile fields when it consumes that config.

## Config Sections Quick Reference

| Section | What It Controls |
|---------|-----------------|
| `networkOperator` | Operator namespace, version, image repository, Helm repository, and `skipHelmChart` ownership switch |
| `docaDriver` | OFED/DOCA driver image, version, blacklist settings |
| `maintenance` | Maintenance Operator, SR-IOV drain, and legacy OFED upgrade concurrency |
| `nvIpam` | NV-IPAM IP pool ranges and subnet generation |
| `sriov` | VF count, resource prefix, MTU, link type |
| `hostdev` | Host device resource name |
| `rdmaShared` | RDMA shared device resource name |
| `ipoib` | IPoIB master interface, resource name |
| `macvlan` | MacVLAN master interface, mode |
| `nicConfigurationOperator` | NIC firmware template settings |
| `spectrumX` | OVS bridge config, multiplane mode, RDMA settings |
| `profile` | Profile selection criteria (fabric, deployment, multirail) |
| `clusterConfig[]` | Per-group hardware: NICs, nodes, capabilities, selectors |

Each `clusterConfig[]` entry has these key fields:

- `identifier` — group name (used for `NicNodePolicy` naming). For groups with both `machineType` and `gpuType` resolved, this is the lowercased machine/GPU identity with complete `NVIDIA` segments removed and common machine segments shortened (`ThinkSystem` → `ts`, `PowerEdge` → `pe`), bounded to 30 bytes with balanced component prefixes and a 6-character deterministic hash when needed; otherwise a fallback `group-N`. The Launch Kit machine node label uses the same value.
- `machineType` — server model (e.g. `PowerEdge-XE9680`); populated from `nvidia.com/gpu.machine` label or DMI fallback.
- `gpuType` — GPU SKU (e.g. `NVIDIA-H200`); populated from `nvidia.com/gpu.product` label or `nvidia-smi` fallback. **Note:** this field used to be called `productType` — the rename happened to disambiguate it from the server model. Old `productType:` keys in hand-authored configs must be renamed to `gpuType:`.
- `netplanManaged` — true when any worker has an NVIDIA PF whose current MAC
  is selected by a host netplan `match.macaddress` stanza with a non-empty
  `set-name`. The flag identifies a potential conflict with NCO udev naming;
  host-specific MACs are not persisted. If generation would emit a
  `NicInterfaceNameTemplate` for this group, clean up the affected `set-name`
  stanzas and re-run discovery instead of editing this flag by hand.
- `capabilities.nodes.{sriov,rdma,ib}` — what the underlying hardware supports.
- `pfs[]` — physical function list with PCI address, device ID, RDMA device,
  network interface, traffic class, rail, NUMA, GPU affinity, and `model` (the
  VPD model/description string read from `NicDevice.Status.modelName`).
  - **One rail per NIC (default).** Discovery advertises one rail per physical NIC: a NIC's multi-plane east-west PFs (planes of one port, e.g. Spectrum-X ConnectX-8/9) collapse to the master PF, so an 8-PF node lists 4 east-west PFs / 4 rails. A NIC whose `model` is genuinely dual-port (`2-port`/`Dual-port`) keeps a rail per port. Run `l8k discover --collapse-nic-rails=false` to emit one rail per PF (legacy/dev behaviour).
- `nodeSelector` — Kubernetes node selector for this group. Source groups key on the machine label written by `l8k discover`: `nvidia.kubernetes-launch-kit.machine: <identifier>`. Auto-merged groups (different machineTypes sharing a GPU type) key on `nvidia.kubernetes-launch-kit.gpu: <gpuType>` instead — the GPU label retains its discovered value, including `NVIDIA`, so the merged selector binds correctly across source machineTypes.
- `workerNodes` — explicit hostnames (populated by discovery).

For the full field-by-field reference with types, defaults, and descriptions,
read `references/config-reference.md`.

## Topology Presets

The `presets/` directory contains pre-recorded topologies for known hardware combinations. A preset is a `topology.yaml` file with the following shape:

```yaml
machineType: PowerEdge-XE9680   # required
gpuType: NVIDIA-H200            # required — both keys are matched as a pair
nicModel: BlueField-3 SuperNIC (ConnectX-7)
gpuInterconnect: NV18
numaNodes: 2

# Required if the preset will be used with `l8k generate --for`.
# Discovery-time overlay does not need this block.
capabilities:
  nodes:
    sriov: true
    rdma: true
    ib: false

pfs:
  - deviceID: a2dc
    pciAddress: 0000:1a:00.0
    traffic: east-west
    rail: 0
    numaNode: 0
    connectedGPU: GPU0
    gpuProximity: PIX
```

**Lookup is exact-match on `(machineType, gpuType)`.** No any-GPU fallback — a preset that doesn't declare `gpuType:` is rejected at load time. Multi-variant presets for the same machine (different GPU SKUs) live in separate directories with composite names like `PowerEdge-XE9680-H200` / `PowerEdge-XE9680-B200`. The directory name is shown by `l8k preset list` and is what `l8k generate --for <name>` accepts.

**Validation deviations.** When the matched preset's PF count, PCI addresses, or device IDs don't exactly match discovered hardware, the preset is **NOT applied** — discovery keeps the live-discovered topology (traffic/rail/NUMA), because overlaying a preset onto a different PCI layout would corrupt the classification (a coincidentally-overlapping PCI address would inherit the preset's unrelated traffic/rail/GPU fields). The discrepancies are recorded under `clusterConfig[*].presetDeviation` and every subsequent config load re-emits a warning listing each deviation. The preset's authoritative topology is overlaid **only on an exact match** (zero deviations); `presetApplied: true` appears only in that case. Part-number and PSID differences are expected (firmware/SKU variants) and never block application.

## Common Edits

```yaml
# Change VF count per PF
sriov:
  numVfs: 16

# Change MTU
sriov:
  mtu: 9000

# Set DOCA driver version
docaDriver:
  version: "doca3.3.0-26.01-1.0.0.0-6"

# Allow four simultaneous maintenance operations. Network Operator 26.1+
# uses the global Maintenance Operator limits; older releases use the legacy
# SR-IOV/OFED limits where applicable.
maintenance:
  maxParallelOperations: 4
  maxUnavailable: 4
  maxNodeMaintenanceTimeSeconds: 3600
  maxParallelUpgrades: 4

# Override the helm chart repository URL (rarely needed — the embedded
# release catalog supplies the right URL for each MAJOR.MINOR release).
# Useful only for mirrors or private chart hosts.
networkOperator:
  helmRepoURL: "https://my-mirror.example.com/charts"
  # Keep generating/applying Network Operator CRs but let another system own
  # the Helm release. Generate/deploy/validate have the equivalent CLI flag
  # --skip-network-operator-helm; clean reads this persistent ownership setting
  # and retains the release while deleting Network Operator CRs.
  skipHelmChart: true

# Configure NV-IPAM subnets manually
nvIpam:
  subnets:
    - name: "rail-0-subnet"
      cidr: "10.10.0.0/16"
      gateway: "10.10.0.1"

# Namespaces for the secondary-network CRs + example test DaemonSets.
# One independent copy is rendered per namespace (shared resources like
# IPPools and NodePolicies are NOT duplicated). Defaults to ["default"].
networkNamespaces: ["my-namespace"]
```

## Tips

- Start by running discovery (`l8k discover`) to generate a baseline, then edit it.
- A discovered config can be passed directly to `l8k generate` without
  repeating profile flags; use flags only for overrides.
- Use `l8k schema` to discover the Network Operator release keys supported by
  the installed l8k version.
- Treat a GA catalog entry as sticky during nightly synchronization: an
  upstream beta or release candidate for the next patch must not replace its
  public Network Operator artifact set. Update the entry again when the patch
  is GA.
- `nvIpam` subnets are auto-generated if not specified — one per rail using non-routable ranges.
- `docaDriver.unloadThirdPartyRDMAModules: true` auto-populates `UNLOAD_THIRD_PARTY_RDMA_MODULES` from discovered OFED-dependent modules.
- For release 26.1+, SR-IOV requestor mode requires both the Network Operator drain requestor and the SR-IOV external drainer. l8k renders both; applying only CRs cannot enable their Deployment environment variables.
- Updating an existing release to the generated requestor-mode Helm values requires `--overwrite-existing`.

## See Also

- [k8s-launch-kit-shared](../k8s-launch-kit-shared/SKILL.md) — Global flags
- [k8s-launch-kit-discover](../k8s-launch-kit-discover/SKILL.md) — Generate a config from live cluster
- `references/config-reference.md` — Complete annotated YAML, including maintenance value restrictions