Aicr Analyzing Snapshots

Analyzing AICR Snapshots

Systematic analysis of AICR snapshot YAML files to extract cluster identity, provider characteristics, GPU topology, node health, software stack, and operational signals. Produces a structured Markdown report.

When to Use

• User provides a snapshot YAML file for review
• User asks about cluster characteristics or provider differentiation
• User wants to compare snapshots or extract specific insights
• User asks to generate a cluster assessment report

Analysis Procedure

Snapshot files are large (50K-80K+ tokens). Never read the whole file. Use mcp__plugin_context-mode_context-mode__execute_file with Python/YAML parsing to extract sections, or use targeted Read with offset/limit on specific line ranges found via Grep.

Step 1: Extract Metadata and Structure

import yaml
data = yaml.safe_load(FILE_CONTENT)
meta = data.get('metadata', {})
measurements = data.get('measurements', [])
print("=== METADATA ===")
for k, v in meta.items():
    print(f"  {k}: {v}")
print("\n=== MEASUREMENTS ===")
for m in measurements:
    subtypes = [s.get('subtype', s.get('name', '?')) for s in m.get('subtypes', [])]
    print(f"  {m['type']}: {subtypes}")

Step 2: Extract K8s Server and Node Info

Key fields for provider identification:

Provider detection logic:

If provider-id is absent, check K8s.server.version for vendor substrings.

Step 3: Extract GPU Info

Key fields from GPU.smi:

Accelerator mapping (checked in order, case-insensitive):

Step 4: Extract OS Info

From OS.release: ID, VERSION_ID, PRETTY_NAME

From OS.grub: Boot parameters (check for iommu, console, init_on_free)

From OS.kmod: Loaded kernel modules (look for nvidia*, nv_peer_mem, gdrdrv, ib_*, mlx5_* for RDMA/InfiniBand)

From OS.sysctl (key tuning parameters):

Step 5: Extract Node Topology

From NodeTopology.summary: node-count, taint-count, label-count

From NodeTopology.taint and NodeTopology.label, read the items list — one entry per distinct reading, sorted by key/value (taints: key/effect/value):

Current snapshots also carry the older data map on both subtypes; items is authoritative. Take counts from data.node-count rather than splitting node-list, and read data.truncated rather than probing for a (+N more) suffix. Use topology.LabelReadings / TaintReadings to resolve items into hydrated readings — they expand node-list-ref automatically, so callers do not need to implement the reference logic themselves.

Older snapshots (no items): fall back to the folded data map — effect|value|node1,node2,... for taints, value|node1,node2,... for labels. That encoding is lossy, so qualify anything derived from it:

• A map key is ambiguous: when a key carries more than one value the value is folded into the key as <key>.<value>, indistinguishable from a label literally named that, and one of the colliding readings is dropped. Report such a key verbatim instead of asserting a key/value split.
• A taint key disambiguated the same way ends in .<effect> and its value has only two fields (value|nodes); two taints sharing key and effect collapse into one entry.
• summary.taint-count / label-count count map entries there, so they under-report wherever a collapse occurred, and node counts reflect only what survived truncation.

High-value labels to extract (skip feature.node.kubernetes.io/cpu-cpuid.*):

Step 6: Extract K8s Images and Policies

From K8s.image: All deployed container images and versions.

From K8s.policy: Flattened GPU Operator ClusterPolicy spec (dot-notation).

Key policy fields:

Step 7: Extract Slinky and MariaDB Conflict Signals

From K8s.slinky-slurm, report:

• collection-state: absent, detected, unsupported-multicluster, or unknown
• Controller count and projected NodeSet/LoginSet/RestApi/Accounting counts
• Item identities and Controller associations; include only the allowlisted item data already present in the snapshot

detected means a Controller declaration exists, not that Slurm or its operator is healthy. Child items and counts are emitted only after all required APIs and references are collected conclusively; their absence is otherwise not confirmed absence. Never infer platform: slurm from this subtype.

From K8s.mariadb-operator, report collection-state as official MariaDB-operator API conflict evidence:

• absent: official API group conclusively absent
• api-detected: official API footprint present without observed MariaDB CRs
• crs-detected: one or more official MariaDB CRs observed
• unknown: discovery or List was inconclusive

These states do not prove database availability, operator health, or the existence of an external database such as RDS. Never infer accounting.databaseSource.

Step 8: Check SystemD Services

From SystemD.containerd.service, SystemD.kubelet.service, SystemD.docker.service:

Report Template

Structure the output as:

# Snapshot Analysis: {name}
> Source: {file} | Captured: {timestamp} | AICR: {version}

## Cluster Identity
Table: source-node, provider, K8s version, node count, GPU model, total GPUs

## Provider-Differentiating Insights
### 1. Provider Type (cloud vs bare-metal, managed vs self-managed)
### 2. CPU Architecture (homogeneous vs heterogeneous, ARM vs x86)
### 3. GPU Hardware (model, architecture, memory, driver, CUDA, MIG, persistence)
### 4. Network Topology (NVLink blocks, cliques, compute domains, RDMA, SR-IOV)
### 5. Management Layer (K8SaaS, NVSentinel health, cordon state)
### 6. Job Scheduling (Slurm/Slinky presence, HPC vs cloud-native)
### 7. Networking Stack (CNI, RDMA, SR-IOV, DOCA/MOFED)
### 8. Security (Confidential Computing, PSA, DRA)
### 9. Operational Signals (sysctl tuning, hugepages, persistence mode)

## Software Stack
### Key Container Images (table)
### OS and Kernel (table)

## Node Inventory
List nodes by rack/block/pool

## Operational Flags
Anything unusual: GPU health issues, disabled persistence mode,
missing hugepages, NVSentinel remediation failures, etc.

What Makes Each Provider Unique

Cloud Providers (EKS, GKE, AKS, OKE)

• Provider-id with cloud prefix
• Cloud-specific K8s version suffixes
• Managed node groups / auto-scaling
• No bare-metal labels (metal3.io)
• Typically x86_64 homogeneous
• No NVLink fabric topology labels
• No Slurm/Slinky stack

Bare-Metal / DGX Cloud (Metal3, K8SaaS)

• metal3:// provider-id with per-node UUIDs
• k8saas.nvidia.com/* management labels
• NVSentinel health monitoring (cordon/uncordon lifecycle)
• NVLink accelerator blocks and GPU cliques
• Compute domains spanning racks
• ARM64 Grace CPUs (heterogeneous with x86 head node)
• Slurm/Slinky HPC scheduling
• RDMA + SR-IOV networking with DOCA drivers
• ATS GPU addressing mode (unified memory)
• Liquid-cooled chassis machine types (LCC in machine name)

Self-Managed / Kind

• Missing or generic provider-id
• No cloud or bare-metal management labels
• Simpler topology (single node or small cluster)
• Standard x86_64

AICR Criteria Mapping

After analysis, map the snapshot to AICR recipe criteria:

aicr recipe \
  --service {detected_service} \
  --accelerator {detected_accelerator} \
  --os {detected_os} \
  --intent {training|inference} \
  --snapshot {snapshot_file}