# K8S Launch Kit Discover

> Use this skill when the user wants to discover their Kubernetes cluster's network hardware capabilities using k8s-launch-kit (l8k). Activate for: cluster discovery, hardware detection, NIC detection, finding what GPUs or NICs are in a cluster, creating a cluster config file, or when the user says 'discover' in the context of l8k or NVIDIA networking.

- Skill: `nvidia/k8s-launch-kit-discover` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add nvidia/k8s-launch-kit-discover`
- Raw SKILL.md: https://api.skillmd.com/api/skills/nvidia/k8s-launch-kit-discover/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Product & Planning
- Author: NVIDIA (https://skillmd.com/u/nvidia)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/nvidia/k8s-launch-kit-discover

---


# l8k: Cluster Discovery

> **PREREQUISITE:** Read `../k8s-launch-kit-shared/SKILL.md` for install paths, global flags, and output modes.

This workflow uses the default `host` target; `--target host` is equivalent.
The CLI snapshots discovery arguments and runs the concrete Host discovery
operation through the target registry; command syntax and artifacts are
unchanged.

Discover cluster hardware and produce a `cluster-config.yaml` describing NICs, GPUs, rails, and node groups.

Three things to know about the saved file:
- **Fresh discovery persists a resolved profile.** Without `--user-config`,
  discovery fills profile fields from hardware/defaults and applies explicit
  CLI overrides last. With `--user-config`, it replaces only `clusterConfig`,
  preserves every other section, and then applies explicit CLI overrides.
- **North-south-only groups are dropped.** A node group whose NICs are all north-south (BlueField DPUs / OOB NICs) produces no manifests and is omitted from `cluster-config.yaml` (it would otherwise consume an NV-IPAM subnet slice and shift the per-rail pools). North-south NICs alongside east-west NICs in a mixed group are still listed.
- **Documentation comments are preserved.** The field-reference comments from the source config (`l8k-config.yaml` or `--user-config`) are carried over into the saved `cluster-config.yaml`, with a `Generated by l8k discover` banner at the top.

**l8k discover is self-contained.** It does NOT require a pre-installed
Network Operator. On every run it bootstraps the NIC Configuration Daemon
(and the 5 nic-configuration-operator CRDs, only if missing) into a private
namespace `nvidia-k8s-launch-kit`, reads `NicDevice` CRs published by the
daemon, then tears the namespace down on exit (unless `--keep-namespace`
is set). The daemon image is pulled from `<networkOperator.repository>/
nic-configuration-operator-daemon:<networkOperator.componentVersion>` (both
fields come from `l8k-config.yaml`).

## Usage (from AI agent)

```bash
l8k discover \
  --kubeconfig ~/.kube/config \
  --save-cluster-config ./cluster-config.yaml \
  --output json 2>/dev/null | jq .
```

## Usage (human-interactive)

```bash
l8k discover --save-cluster-config <OUTPUT> [--kubeconfig <PATH>]
```

## Flags

| Flag | Required | Default | Description |
|------|----------|---------|-------------|
| `--kubeconfig` | — | `$KUBECONFIG` env var | Path to kubeconfig (optional — falls back to env var) |
| `--save-cluster-config` | Yes | — | Output path for cluster-config.yaml |
| `--keep-namespace` | — | `false` | Skip teardown of the `nvidia-k8s-launch-kit` bootstrap namespace (for debugging) |
| `--collapse-nic-rails` | — | `true` | Advertise one rail per NIC: collapse a NIC's multi-plane east-west PFs to its master PF, keeping a rail per port only for NICs whose VPD model is genuinely dual-port ("2-port"/"Dual-port"). Set `=false` to keep one rail per PF (dev setups). See "Rail collapsing" below. |
| `--network-operator-namespace` | — | — | **Deprecated for `discover`**: accepted but ignored. The daemon always runs in `nvidia-k8s-launch-kit`. Still used by `l8k generate` / `l8k deploy`. |
| `--user-config` | — | — | Config whose `clusterConfig` is refreshed; all other sections are preserved unless explicitly overridden by a CLI flag |
| `--node-selector` | — | `feature.node.kubernetes.io/pci-15b3.present=true` | Value written into the **saved** `cluster-config.yaml` `nodeSelector` (for deploy time). It does **not** gate discovery scheduling or the NicDevice wait set — the daemon is restricted to Ready schedulable nodes and discoverable NICs are detected via sysfs; restricted BlueFields are excluded. |
| `--image-pull-secrets` | — | — | Image pull secret names (comma-separated). Forwarded to the discovery DaemonSet and persisted for generated policies, Network Operator Helm values, and authenticated chart downloads during deploy. |
| `--fabric` | — | discovered unanimous link type | Fabric override: `ethernet` or `infiniband`. |
| `--deployment-type` | — | `sriov` | Deployment override: `sriov`, `rdma_shared`, or `host_device`. |
| `--multirail` | — | `true` | Multirail override. Use `--multirail=false` to opt out; explicit false is persisted. |
| `--spectrum-x` | — | disabled | Enable Spectrum-X with an RA version (`RA2.1` or `RA2.2`). |
| `--multiplane-mode` | — | derived with Spectrum-X | Spectrum-X mode override: `none`, `swplb`, or `hwplb`. |
| `--number-of-planes` | — | derived with Spectrum-X | Spectrum-X plane-count override: `1`, `2`, or `4`. |

## Examples

```bash
# Basic discovery
l8k discover \
  --kubeconfig ~/.kube/config \
  --save-cluster-config ./cluster-config.yaml

# Using $KUBECONFIG env var (no --kubeconfig needed)
l8k discover --save-cluster-config ./cluster-config.yaml

# Keep the bootstrap namespace for debugging
l8k discover \
  --kubeconfig ~/.kube/config \
  --keep-namespace \
  --save-cluster-config ./cluster-config.yaml

# Refresh clusterConfig in an existing config
l8k discover --user-config my-config.yaml \
  --kubeconfig ~/.kube/config \
  --save-cluster-config ./cluster-config.yaml

# Persist explicit profile overrides
l8k discover \
  --kubeconfig ~/.kube/config \
  --fabric infiniband --deployment-type rdma_shared \
  --multirail=false \
  --save-cluster-config ./cluster-config.yaml

# Agent mode (JSON output)
l8k discover \
  --kubeconfig ~/.kube/config \
  --save-cluster-config ./cluster-config.yaml \
  --output json 2>/dev/null
```

## Output Format

The generated `cluster-config.yaml` contains the final profile plus a
`clusterConfig[]` hardware-group array:

```yaml
profile:
  fabric: ethernet
  deployment: sriov
  multirail: true
clusterConfig:
  - identifier: "dgx-b200-h100-nvl"
    machineType: DGX-B200
    gpuType: NVIDIA-H100-NVL
    netplanManaged: true
    capabilities:
      nodes:
        sriov: true
        rdma: true
    pfs:
      - deviceID: "101e"
        networkInterface: "eth0"
        rail: 0
    workerNodes: [node-01, node-02]
    nodeSelector:
      nvidia.kubernetes-launch-kit.machine: "dgx-b200-h100-nvl"
    thirdPartyRDMAModules: [nv_peer_mem]
```

Discovery patches every node in the group with two labels:

- `nvidia.kubernetes-launch-kit.machine: <identifier>` — per-source-group
  identity, identical to the persisted group `identifier` and used as its
  `nodeSelector`.
- `nvidia.kubernetes-launch-kit.gpu: <gpuType>` — used as the merged-group
  `nodeSelector` when `l8k generate` auto-merges source groups sharing a GPU type.

Group identifiers and machine-label values are lowercase and omit complete `NVIDIA`
segments. The same pass maps `ThinkSystem` to `ts` and `PowerEdge` to `pe`. They are
bounded to 30 bytes; long values retain balanced machine/GPU prefixes plus a
6-character deterministic hash, and unused prefix space moves to the longer component.
GPU-only labels retain their discovered case and vendor segment, use Kubernetes'
63-byte limit, and keep the existing 8-character shortening hash. When `machineType`
or `gpuType` couldn't be resolved (GPU operator labels absent and hardware probe
failed), a fallback `group-N` identifier is used and the machine label is not written;
the GPU label is still written when `gpuType` alone is resolved.

## Prerequisites

- Node Feature Discovery (NFD) is **not** required. Before bootstrapping the
  daemon, discovery lists Nodes and renders node-name affinity for nodes that
  are `Ready=True` and not `spec.unschedulable`. The DaemonSet still has no
  NFD `nodeSelector` and tolerates taints, so eligible control-plane/tainted
  nodes are included. NIC-bearing nodes are detected by a sysfs probe for PCI
  vendor `0x15b3` rather than the NFD
  `feature.node.kubernetes.io/pci-15b3.present` label. BlueFields for which
  `mlxprivhost` reports `level: restricted` are excluded because the daemon
  will not publish `NicDevice` resources for them. Mixed nodes remain eligible
  when they have at least one non-restricted NVIDIA NIC.
- Image-pull access from every worker node to
  `<networkOperator.repository>/nic-configuration-operator-daemon:<networkOperator.componentVersion>`.
  Use `--image-pull-secrets <name>` (or `networkOperator.imagePullSecrets`
  in the config file) for private registries.
- Network Operator does **NOT** need to be pre-installed.

Fresh discovery writes `validation.gpuDirect.enabled`. It is true only when
every worker in every discovered group can satisfy its render bucket's
topology-derived `validation.gpuDirect.gpuResourceType` request; otherwise it
is false. With `--user-config`, the supplied validation section is preserved.
Do not add GPU counts or resource maps under `clusterConfig`.

## Tips

- Fresh profile precedence is `hardware/default < explicit CLI`. On a rerun
  with `--user-config`, no profile fields are recalculated; explicit CLI flags
  still override their corresponding values.
- A fresh discovery ignores the example profile in the installed reference
  config. Supplying that file explicitly through `--user-config` makes its
  profile values user input and therefore preserves them.
- If fabric probes are mixed or unconfirmed, discovery still writes the file
  with an empty `profile.fabric`; set `--fabric` on discovery or generation.
- `clusterConfig[].netplanManaged` is true when any worker has an NVIDIA PF
  selected by a host netplan `match.macaddress` plus `set-name` rule. Discovery
  checks every worker and uses PF MACs only transiently; they are not saved in
  the shared group config. If generation would emit a
  `NicInterfaceNameTemplate` for an affected group, clean up the host's
  `set-name` stanzas and re-run discovery before retrying generation; do not
  clear the field manually.

- The bootstrap namespace is **always** `nvidia-k8s-launch-kit` — not configurable. The
  daemon's SA / ClusterRole / ClusterRoleBinding are renamed to
  `k8s-launch-kit-nic-config-daemon` so they don't collide with the cluster-scoped
  names a coexisting Network Operator install would create.
- CRDs (`nicdevices`, `nicconfigurationtemplates`, etc.) are applied **only when missing**.
  If they already exist (because Network Operator or a prior l8k discover run created
  them), they're left alone — discovery never overwrites a different version.
- After discovery finishes the namespace is torn down (cascade-delete handles SA /
  RoleBinding / DaemonSet / pods / NicDevice CRs); the CRDs intentionally persist so
  any external NicDevice consumers survive. Cluster-scoped ClusterRole /
  ClusterRoleBinding are deleted explicitly.
- Pass `--keep-namespace` to leave the namespace in place — useful when debugging
  daemon pod start-up failures (`kubectl describe pod -n nvidia-k8s-launch-kit`).
- Before bootstrapping, discovery pre-cleans the `nvidia-k8s-launch-kit` namespace
  (deletes any leftover DaemonSet/pods/NicDevice CRs from a crashed prior run and
  waits up to 2 min for it to clear) so a fresh daemon is never layered on stale state.
- Discovery waits up to 5 minutes for daemon pods to be Ready, but tolerates
  stuck pods: if some pods are wedged (e.g. ImagePullBackOff/CrashLoopBackOff on
  an unrelated node) it proceeds with the Ready ones rather than blocking the
  whole window, and only aborts if **no** pod ever becomes Ready. Common causes
  of a total failure: image tag missing in your registry, pull-secret missing.
- If discovery reports "no nodes with an NVIDIA NIC (PCI vendor 15b3) were found",
  the daemon ran but no node's sysfs exposed a `0x15b3` device (no Mellanox/NVIDIA
  NICs, or `/sys` not mounted/readable in the pod).
- If discovery reports "no discoverable NVIDIA NICs were found", every detected
  BlueField is in zero-trust (`restricted`) mode. Those devices cannot be
  configured from the host and intentionally do not publish `NicDevice` resources.
- After determining each group's `(machineType, gpuType)`, discovery looks up a topology preset under `presets/` using **exact-match** lookup on that pair. A matching preset overrides heuristic-derived topology fields (traffic class, rail, NUMA, GPU affinity). There is no any-GPU fallback — a preset with empty `gpuType:` is rejected at load time. If no preset matches, discovery proceeds with heuristic classification.
- If you already know the SKU and want to skip cluster discovery entirely, use `l8k generate --for <preset>` (see `k8s-launch-kit-generate`).

## Rail collapsing (one rail per NIC)

By default (`--collapse-nic-rails`, default `true`) discovery advertises **one
rail per NIC**, not one per PF. A single NIC can expose several east-west PFs
that are *planes* of one physical port (Spectrum-X multi-plane ConnectX-8/9). For
those, only the **master PF** (lowest PCI function) is written to
`cluster-config.yaml`, so the rail count reflects physical NICs — e.g. an 8-PF
ConnectX-8 node yields 4 rails, not 8. The per-PF planes are reconstructed at
generate time from the multiplane mode + `numberOfPlanes`.

The exception is a **genuinely dual-port NIC**: when the NIC's VPD model name
(read from `NicDevice.Status.modelName`, surfaced as the `model` field on each
PF) contains a port-count keyword like `2-port`/`Dual-port`, each port is kept as
its own rail. Single-port (`1P`) and multi-plane models (no port-count keyword)
collapse. An empty/unreadable model collapses by default.

Pass `--collapse-nic-rails=false` to restore the legacy one-rail-per-PF behaviour
(handy on dev setups). Note: a collapsed group no longer has the same PF count as
a full-PF topology preset, so such presets won't apply by exact match and the
live (collapsed) classification is kept (a preset-deviation warning is emitted).

## See Also

- [k8s-launch-kit-shared](../k8s-launch-kit-shared/SKILL.md) — Global flags and output modes
- [k8s-launch-kit-config](../k8s-launch-kit-config/SKILL.md) — Understand the config file discovery produces
- [k8s-launch-kit-generate](../k8s-launch-kit-generate/SKILL.md) — Generate manifests from cluster config