Nvidia Aicr Aicr

Analyzing AICR Snapshots

Systematic analysis of AICR snapshot YAML files to extract cluster identity, provider characteristics, GPU topology, node health, software stack, and operational signals. Produces a structured Markdown report.

When to Use

• User provides a snapshot YAML file for review
• User asks about cluster characteristics or provider differentiation
• User wants to compare snapshots or extract specific insights
• User asks to generate a cluster assessment report

Analysis Procedure

Snapshot files are large (50K-80K+ tokens). Never read the whole file. Use mcp__plugin_context-mode_context-mode__execute_file with Python/YAML parsing to extract sections, or use targeted Read with offset/limit on specific line ranges found via Grep.

Step 1: Extract Metadata and Structure

import yaml
data = yaml.safe_load(FILE_CONTENT)
meta = data.get('metadata', {})
measurements = data.get('measurements', [])
print("=== METADATA ===")
for k, v in meta.items():
    print(f"  {k}: {v}")
print("\n=== MEASUREMENTS ===")
for m in measurements:
    subtypes = [s.get('subtype', s.get('name', '?')) for s in m.get('subtypes', [])]
    print(f"  {m['type']}: {subtypes}")

Step 2: Extract K8s Server and Node Info

Key fields for provider identification:

Provider detection logic:

If provider-id is absent, check K8s.server.version for vendor substrings.

Step 3: Extract GPU Info

Key fields from GPU.smi:

Accelerator mapping (checked in order, case-insensitive):

Step 4: Extract OS Info

From OS.release: ID, VERSION_ID, PRETTY_NAME

From OS.grub: Boot parameters (check for iommu, console, init_on_free)

From OS.kmod: Loaded kernel modules (look for nvidia*, nv_peer_mem, gdrdrv, ib_*, mlx5_* for RDMA/InfiniBand)

From OS.sysctl (key tuning parameters):

Step 5: Extract Node Topology

From NodeTopology.summary: node-count, taint-count, label-count

From NodeTopology.taint: Format is effect|value|node1,node2,...

From NodeTopology.label: Format is value|node1,node2,...

High-value labels to extract (skip feature.node.kubernetes.io/cpu-cpuid.*):

Step 6: Extract K8s Images and Policies

From K8s.image: All deployed container images and versions.

From K8s.policy: Flattened GPU Operator ClusterPolicy spec (dot-notation).

Key policy fields:

Step 7: Check SystemD Services

From SystemD.containerd.service, SystemD.kubelet.service, SystemD.docker.service:

Report Template

Structure the output as:

# Snapshot Analysis: {name}
> Source: {file} | Captured: {timestamp} | AICR: {version}

## Cluster Identity
Table: source-node, provider, K8s version, node count, GPU model, total GPUs

## Provider-Differentiating Insights
### 1. Provider Type (cloud vs bare-metal, managed vs self-managed)
### 2. CPU Architecture (homogeneous vs heterogeneous, ARM vs x86)
### 3. GPU Hardware (model, architecture, memory, driver, CUDA, MIG, persistence)
### 4. Network Topology (NVLink blocks, cliques, compute domains, RDMA, SR-IOV)
### 5. Management Layer (K8SaaS, NVSentinel health, cordon state)
### 6. Job Scheduling (Slurm/Slinky presence, HPC vs cloud-native)
### 7. Networking Stack (CNI, RDMA, SR-IOV, DOCA/MOFED)
### 8. Security (Confidential Computing, PSA, DRA)
### 9. Operational Signals (sysctl tuning, hugepages, persistence mode)

## Software Stack
### Key Container Images (table)
### OS and Kernel (table)

## Node Inventory
List nodes by rack/block/pool

## Operational Flags
Anything unusual: GPU health issues, disabled persistence mode,
missing hugepages, NVSentinel remediation failures, etc.

What Makes Each Provider Unique

Cloud Providers (EKS, GKE, AKS, OKE)

• Provider-id with cloud prefix
• Cloud-specific K8s version suffixes
• Managed node groups / auto-scaling
• No bare-metal labels (metal3.io)
• Typically x86_64 homogeneous
• No NVLink fabric topology labels
• No Slurm/Slinky stack

Bare-Metal / DGX Cloud (Metal3, K8SaaS)

• metal3:// provider-id with per-node UUIDs
• k8saas.nvidia.com/* management labels
• NVSentinel health monitoring (cordon/uncordon lifecycle)
• NVLink accelerator blocks and GPU cliques
• Compute domains spanning racks
• ARM64 Grace CPUs (heterogeneous with x86 head node)
• Slurm/Slinky HPC scheduling
• RDMA + SR-IOV networking with DOCA drivers
• ATS GPU addressing mode (unified memory)
• Liquid-cooled chassis machine types (LCC in machine name)

Self-Managed / Kind

• Missing or generic provider-id
• No cloud or bare-metal management labels
• Simpler topology (single node or small cluster)
• Standard x86_64

AICR Criteria Mapping

After analysis, map the snapshot to AICR recipe criteria:

aicr recipe \
  --service {detected_service} \
  --accelerator {detected_accelerator} \
  --os {detected_os} \
  --intent {training|inference} \
  --snapshot {snapshot_file}

Source: NVIDIA/aicr — distributed by TomeVault.