# Job Analysis

> Analyze an HPC job from an Omnistat database using hypothesis-driven exploration, driven by the omnistat-inspect tool. Use this to diagnose why a job behaved as it did — performance bottlenecks, hardware issues, anomalies, or comparing a degraded job against a healthy baseline. For a plain factual snapshot without investigation, use job-report instead.

- Skill: `rocm/job-analysis` (Agent Skill, multi-file: 3 files)
- Install (CLI): `npx skillmds@latest add rocm/job-analysis`
- Raw SKILL.md: https://api.skillmd.com/api/skills/rocm/job-analysis/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Docs & Writing
- Author: rocm (https://skillmd.com/u/rocm)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/rocm/job-analysis

---


# Job Analysis

Analyze GPU telemetry data collected by Omnistat for HPC/AI workloads. This skill guides you through a top-down, hypothesis-driven analysis of job performance, driven primarily by the `omnistat-inspect` CLI tool.

**Target audience:** HPC engineers, AI/ML researchers, system administrators investigating job performance, GPU health, and resource utilization.

**What the analysis produces:** A structured report identifying performance bottlenecks, hardware issues, resource utilization patterns, and anomalies -- with all findings backed by data.

**When to use this vs `job-report`.** Use **job-analysis** when you need to understand *why* a job behaved as it did — diagnosing bottlenecks, throttling, stragglers, or regressions, or comparing a degraded job against a healthy baseline. It is hypothesis-driven and iterative. If you only need a quick factual snapshot of *what* a job did (stats, energy, health, data quality) without investigation, use **job-report** instead. A common pattern is to run job-report first, then reach for job-analysis when something looks off.

## Tooling: omnistat-inspect

This skill is built entirely around `omnistat-inspect`, the consolidated analysis CLI. It is your single entry point for every phase:

- **Baseline characterization** — `omnistat-inspect --tsdb-url $TSDB_URL job JOBID report` produces the structured report card (overview, stats, variance, data-collection, health). Start every analysis here: it is the fastest way to understand scale, runtime, utilization, variance, and health in a single call.
- **Deep-dive subcommands** — `job JOBID info` (metadata/topology), `stats` (gauges, counters, hardware counters, and per-node/per-GPU variance), `health` (data-collection coverage and health checks), `iterations` (iteration boundaries and per-iteration stats), `query` (arbitrary PromQL), and `timeseries` (raw series export).
- **Data-source inspection** — `omnistat-inspect --tsdb-url $TSDB_URL db info` lists the jobs and metrics available in the backend (no job context required).

For anything not covered by a subcommand, drop to raw PromQL via `query` (TSDB) or `curl` against the TSDB HTTP API.

### Job-context flexibility

Every `omnistat-inspect job JOBID` invocation resolves the job's time window in one of two ways:

- **Discovery (default):** omits `--start`/`--end`; the tool scans the database to discover the job's time range and topology. Add `--cache-dir DIR` to persist the discovery snapshot and per-section results so repeat calls are cheap (no re-scan, no re-query).
- **Direct window:** pass both `--start ISO8601` and `--end ISO8601` (optionally `--interval SECONDS`) to skip discovery entirely and analyze an exact window — useful for zooming into a single phase or iteration you found earlier.

```bash
# Discovery + cache (cheap repeat calls)
omnistat-inspect --tsdb-url $TSDB_URL --cache-dir $SCRATCH/cache job JOBID report

# Direct window (no discovery scan)
omnistat-inspect --tsdb-url $TSDB_URL job JOBID \
  --start 2026-01-01T12:00:00Z --end 2026-01-01T12:10:00Z report
```

#### Cache-dir reuse for late-pipeline `query` / `timeseries`

The `query` and `timeseries` subcommands are typically run late in the analysis,
well after discovery. **Always pass the same `--cache-dir` you used for the
initial `report`/`info` call** so they rehydrate the cached discovery
snapshot instead of re-scanning:

```bash
# Early: discovery runs once and is cached (time range + sampling interval)
omnistat-inspect --tsdb-url $TSDB_URL --cache-dir $SCRATCH/cache job JOBID info

# Later: query/timeseries reuse the snapshot — no re-scan, correct default step
omnistat-inspect --tsdb-url $TSDB_URL --cache-dir $SCRATCH/cache job JOBID \
  query --promql 'avg(rocm_utilization_percentage{$job, $jobstep})'
```

Because the rehydrated snapshot restores the discovered **sampling interval**,
the default query step is the sampling interval (`max(sampling_interval, 1s)`) —
exactly what you want for full-resolution queries. Notes:

- **Do not** reach for `--start`/`--end` just to "scope to the discovered
  window" — passing them *skips* discovery and drops the sampling interval, so
  the default step silently degrades to **1s** unless you also add `--interval`.
  Use `--start`/`--end` only when you genuinely want a narrower sub-window, and
  pair them with `--interval` to keep the step correct.
- For a deliberately coarser step on a single `query` (e.g. an overview of a
  long job), pass `--step SECONDS` directly; it overrides the default for that
  call only. (`timeseries` has no `--step`; control its resolution via the
  cached interval or the global `--interval`.)

## Prerequisites

1. **Data source** — one of:
   - **VictoriaMetrics running** with the Omnistat database loaded (use the `open-database` skill if needed), OR
   - **CSV exports** from `omnistat-query --export` (no TSDB required)
2. **Python virtual environment activated** with omnistat installed (`pip install ".[query]"` from the omnistat repo root) — this provides `omnistat-inspect`. Confirm with `which omnistat-inspect`.
3. **Job ID(s)** to analyze (discover available jobs with `omnistat-inspect --tsdb-url $TSDB_URL db info` or `omnistat-inspect --csv-dir /path/to/exports db info`)

## Setup

Before starting analysis, set up the working environment:

### TSDB Mode (default)

```bash
# 1. Create a cache directory for this analysis session (cheap repeat calls)
SCRATCH=$(mktemp -d /tmp/omnistat-inspect-XXXXXX)
echo "Cache directory: $SCRATCH/cache"

# 2. Set the TSDB URL (VictoriaMetrics or Prometheus)
TSDB_URL="http://localhost:8428"

# 3. Verify connectivity and discover available jobs
omnistat-inspect --tsdb-url $TSDB_URL db info
```

### CSV Mode

Use CSV mode when you have CSV exports from `omnistat-query --export` and no running TSDB. All subcommands except `job query` work in CSV mode — CSV mode uses whatever metrics were exported, so if a metric wasn't included in the export, it won't be available for analysis.

```bash
# 1. Create a cache directory for this analysis session
SCRATCH=$(mktemp -d /tmp/omnistat-inspect-XXXXXX)
echo "Cache directory: $SCRATCH/cache"

# 2. Set the CSV directory path
CSV_DIR="/path/to/csv/exports"

# 3. Discover available data
omnistat-inspect --csv-dir $CSV_DIR db info

# 4. Run analysis (same subcommands as TSDB mode)
omnistat-inspect --csv-dir $CSV_DIR --cache-dir $SCRATCH/cache job JOBID info
omnistat-inspect --csv-dir $CSV_DIR --cache-dir $SCRATCH/cache job JOBID stats
omnistat-inspect --csv-dir $CSV_DIR --cache-dir $SCRATCH/cache job JOBID health
```

**Note:** The `job query` subcommand (arbitrary PromQL) is not available in CSV mode — it requires a TSDB backend.

The `db info` subcommand verifies database connectivity and reports all available jobs with their time ranges, node counts, users, and partitions, plus the full list of available metrics. Use this output to select a job ID and confirm you are looking at the right database.

## Analysis Workflow

Follow this top-down, hypothesis-driven workflow. Each phase builds on the previous one. You have freedom to explore and investigate -- this is a methodology guide, not a rigid script.

### Epistemic Discipline

**Do not assume what the workload is.** Unless the user tells you the application name, or annotations/metadata explicitly identify it, treat the workload as unknown. Describe what the telemetry shows (e.g., "the GPUs spend ~40% of wall-clock idle between compute phases, each phase ~90s long") rather than what you think it means (e.g., "this is a training workload doing forward/backward passes"). Note: GPU utilization naturally sits near 0% or near 100%, so simply calling it "bimodal" is not an insight — quantify the idle fraction or phase structure instead. If you need to speculate, label it clearly as a hypothesis.

**Do not assume the workload is homogeneous.** A single HPC job may run different tasks on different nodes or GPUs. Some nodes may run data loading, others may run compute, others may handle communication. VRAM differences across GPUs, utilization variance across nodes, or non-uniform network traffic are signals of heterogeneity, not necessarily problems. Before reporting "imbalance" as a finding, consider whether the workload is intentionally heterogeneous.

**Report what you observe, not what you expect.** If a metric looks unusual, describe the observation and its magnitude. Do not assume it is a problem unless you have evidence of impact (e.g., on runtime, throughput, or health). An observation like "5% of GPUs use 10x more VRAM than the rest" is a fact; "there is a memory imbalance problem" is an interpretation that may be wrong.

### Job Discovery and Characterization

**The first step of every analysis is the one-shot report.** It is the factual baseline the rest of the workflow builds on — a single call that returns the job overview, the full `stats` block (gauges, counters, hardware counters, variance), and the `health` block (data-collection coverage + hardware health). Save it and reuse its embedded blocks; the downstream sections below consume this output rather than re-fetching the same data.

```bash
# First step — factual baseline. Embeds overview + stats + health in one JSON.
omnistat-inspect --tsdb-url $TSDB_URL --cache-dir $SCRATCH/cache job JOBID report > $SCRATCH/report_JOBID.json

# List all metrics available in the data source, plus all jobs and time ranges
omnistat-inspect --tsdb-url $TSDB_URL db info
```

The `report` JSON has top-level keys `overview`, `stats`, and `health`. Everything the "Data Collection and Hardware Health Validation" and "Statistical Analysis" sections need is already in this one document — you only issue additional `stats` / `health` / `info` calls to **drill down** (finer grouping), **refresh** (after `--interval` changes), or when you deliberately **skipped** the baseline report. `job info` on its own returns just the overview subset if you ever need it in isolation.

Key information to extract:
- **Runtime**: How long did the job run?
- **Scale**: How many nodes and GPUs?
- **Sampling interval**: What time resolution is available?
- **Available metrics**: Which collectors were active? (GPU, host, network, RAS, xGMI, rocprofiler) — `db info` lists every metric present in the backend.
- **Annotations**: `rmsjob_annotations` markers (e.g., application phases, benchmark identifiers)
- **Figure of Merit**: `omnistat_fom` values (e.g., GFLOPS achieved)

The `job info` subcommand automatically includes `annotations` and `figure_of_merit` when the corresponding metrics are present in the database.

The `job info` subcommand reports the discovered sampling interval (auto-detected from the `omnistat_info` metric's `interval_secs` label). The sampling interval is also used internally by `stats`, `health`, and `iterations` to auto-compute the finest safe query step — you do not need to pass `--interval` to these subcommands.

### GPU Architecture Detection

After discovering the job, identify the GPU architecture from the available metrics and load the corresponding architecture profile for GPU-specific domain knowledge (power reporting quirks, thermal limits, memory characteristics, RAS error blocks, hardware counter formulas).

Architecture profiles are located in `skills/job-analysis/gpus/`. Read the matching profile before proceeding to data-collection and health validation.

**Detection:** Use `overview.gpu_type` from the `job info` output and apply the same substring rules as the job-report skill's "GPU Architecture Handling" table (`MI250` or `MI200 (MCM)` → MI250X; `MI300` → MI300X). When `gpu_type` is a list, apply the rule to each element.

The architecture profile contains critical information for correct interpretation of the data (e.g., which GPU cards report power, thermal throttling thresholds, RAS error block meanings).

### Resolution Sensitivity

Step resolution significantly affects observed statistics. Coarse steps (e.g., 60s) average over intervals, smearing peaks and troughs together. This can be seriously misleading:

- **Peak metrics are underestimated** at coarse resolution (e.g., peak FOM at 60s may be 10-25% lower than at 5s)
- **Mean metrics are mostly unaffected** by resolution (averaging preserves the mean)
- **Iteration boundaries blur** at coarse resolution, making it impossible to distinguish per-iteration behavior

**Always verify critical findings at the finest feasible resolution.** The finest meaningful resolution is the sampling interval reported by `job info` (from `omnistat_info`'s `interval_secs` label) — querying at a finer step than this adds no real data.

#### Step Selection

The `stats`, `health`, and `iterations` subcommands **auto-compute the finest safe query step**. The step is `max(sampling_interval, runtime / 90000)` — never finer than the actual data, never exceeding VictoriaMetrics' `search.maxPointsPerTimeseries` limit (90,000). There is no arbitrary floor: sub-second sampling intervals are preserved for short jobs where VM limits allow it.

`--interval` is a flag on the `job` group and must be placed *before* the subcommand (e.g. `job JOBID --interval N stats`), not after it. The `iterations` subcommand ignores `--interval` — it always uses an auto-computed step. For `stats` and `health` it refines the time range only, while the query step stays auto-computed.

For `timeseries` and `query`, the default step is the discovered sampling interval (`max(sampling_interval, 1s)`) when you reuse the cached discovery snapshot via `--cache-dir` — full resolution with no extra flags. For a coarser overview on a long job, `query` accepts an explicit `--step SECONDS`; `timeseries` has no `--step`, so adjust its resolution via the cached interval or the global `--interval`.

**When the auto-computed step is much coarser than the sampling interval** (which happens on very long jobs), state the resolution gap explicitly in the report and note which findings may be affected (especially peaks and percentiles).

**Critical rule for peak metrics:** If peak FOM, peak utilization, or peak throughput appears degraded, **always re-verify at the finest feasible step** (using `query` with an explicit `--step`) before concluding there is a peak performance difference. Apparent peak degradation is frequently an artifact of temporal averaging — the true peaks may be identical across jobs. Do not claim peak performance differs without checking at fine resolution.

### Data Collection and Hardware Health Validation

Before analyzing performance, verify that data collection was complete and reliable, and check for hardware issues. **This data is already in the baseline report's `health` block — read it from there; do not re-fetch.** Only run `health` standalone if you skipped the baseline report or need to refresh after changing `--interval`:

```bash
# Standalone health (only if you skipped the baseline report or are refreshing)
omnistat-inspect --tsdb-url $TSDB_URL --cache-dir $SCRATCH/cache job JOBID health
```

The `health` block covers both data-collection coverage (completeness, timing stagger, gaps) and hardware health (RAS errors, thermals, power).

#### Data-collection coverage (part of `health`)

Review the coverage portion of the health report for:
- **Missing nodes**: `expected_nodes` vs `reporting_nodes` — any gap means some nodes never reported
- **Activation stagger**: `activation_stagger_seconds` — how long it took for all nodes to start reporting. A spread >5% of total job duration is significant and means early-job statistics are skewed by partial participation
- **Deactivation stagger**: `deactivation_stagger_seconds` — same for shutdown. Large spread means late-job statistics are unreliable
- **Sampling gaps**: `nodes_with_gaps` and `total_gaps` — gaps are reported only as counts, not per-gap timing. To localize them (e.g., distinguish a clustered systemic event from distributed per-node issues), drill down with `timeseries`/`query`
- **Reporting duration**: `reporting_duration_per_node_seconds` (a `{mean, min, max}` object) — nodes with significantly shorter reporting durations may have crashed or been evicted mid-job

#### Hardware health (`health`)

Review the health report for:
- **RAS errors**: Any hardware errors during the job
- **Thermal issues**: GPUs running hot
- **Power anomalies**: Unexpected zero-power readings
- **Push health**: Whether monitoring push duration exceeded the push interval (indicates monitoring overhead)

If critical issues are found, note them -- they may explain performance anomalies found later.

### Statistical Analysis

Follow these steps in order. **Do not skip steps or move to iteration analysis until all steps are complete.**

#### Step 1: Read the baseline stats

The job's `stats` are **already in the baseline report** (`report_JOBID.json` → `stats`): global gauge/counter summaries, hardware counters, and per-node / per-GPU variance, all in one block — no `--category` or `--level` flags exist or are needed. Read them from the baseline; only run `stats` standalone if you skipped the report or are refreshing after an `--interval` change:

```bash
# Standalone stats (only if you skipped the baseline report or are refreshing)
omnistat-inspect --tsdb-url $TSDB_URL --cache-dir $SCRATCH/cache job JOBID stats > $SCRATCH/stats_JOBID.json
```

The `stats` block has keys `gauges`, `counters`, `hardware_counters`, and `variance`. Counter metrics (cumulative values like bytes transferred, energy consumed) are automatically detected and produce delta-based totals. Gauge metrics produce mean/min/max/cv/percentiles. The `cv` (coefficient of variation) field measures relative dispersion — high CV indicates non-uniform distribution across GPUs or nodes.

#### Step 2: Identify anomalous metrics

Review the gauge and counter stats. For each metric, check for:
- High `cv` (uneven distribution across nodes/GPUs)
- Unexpected values (rates, totals, or distributions that differ from expectation)
- Large gaps between percentiles (for GPU utilization specifically, a near-0/near-100 split is the expected norm — not a finding; quantify idle time or phase structure rather than labeling it "bimodal")

**In comparative analysis:** compare each metric between the healthy and degraded jobs. Identify which show significant differences (>10% in rates or totals, >5 percentage points in gauge means).

#### Step 3: Inspect the variance breakdown

The `variance` block in the same `stats` output already drills into any metric whose between-node or between-GPU CV exceeds the threshold (default `cv_threshold=0.05`; override with `--cv-threshold`). Each entry first collapses every spatial key's whole-job series to **one scalar** — the `reduction` field names which one: `temporal_mean` (time-average of samples; plain gauges), `rate` (Δtotal ÷ active duration; counter-derived gauges like network RX/TX), `ratio` (Δa ÷ Δb between two counters; kernel mean dispatch duration = Δduration ÷ Δdispatches), or `total` (a single reset-aware cumulative-counter delta per key). Counter metrics (`COUNTER_LIST`: IO, network, vendor energy, xGMI) now participate in `stats.variance` with `reduction: "total"`, grouped by `source` like gauges — every counter gets a `by_node` entry; GPU-source counters (xGMI) also get `by_gpu`/`by_gpu_id`. It then carries the **between-key `cv`** (dispersion across those per-key reduced values), `min`/`max` (the lowest/highest per-key reduced value and the key that owns it — extremes of per-key reductions, **not** absolute sample minima/maxima, which live in `stats.gauges[].min`/`max`), and either an `all` map listing every key (when `n ≤ 16`) or `percentiles` over the per-key values (when `n > 16`; `--verbose` adds `all` too). The three groupings:
- `by_node` — one reduced value per node (`{instance}`); straggler nodes, systemic vs. node-local effects
- `by_gpu_id` — one per card slot (`{card}`); card-position effects
- `by_gpu` — one per (node, card) (`{instance, card}`); individual GPU stragglers

Use `--verbose` to force full per-entity arrays even for large jobs. To raise sensitivity, lower `--cv-threshold`:

```bash
# More sensitive variance drill-down: run for BOTH jobs
omnistat-inspect --tsdb-url $TSDB_URL --cache-dir $SCRATCH/cache job JOBID stats --cv-threshold 0.02 --verbose > $SCRATCH/stats_verbose_JOBID.json
```

The step is auto-computed to stay within `maxPointsPerTimeseries` limits, so queries should not fail due to point limits. If a query does fail, the `--interval` flag can be used as an override to force a coarser step.

The variance breakdown answers critical questions that the global summary cannot:
- Is the anomaly systemic (all nodes/GPUs equally affected) or localized?
- Is there a straggler node or a systematic card-position effect?

**In comparative analysis:** examine the `variance` block for **both** the healthy and degraded jobs so you can compare at each grouping.

The `variance` block exposes three groupings, from coarse to fine:

| Grouping | Key | What it reveals |
|----------|-----|-----------------|
| Per-node | `by_node` | Straggler nodes; systemic vs. node-local effects |
| Per-card-slot | `by_gpu_id` | Systematic card-position effects (e.g., all card-0s behaving differently) |
| Per-GPU | `by_gpu` | Individual GPU outliers masked by node/slot averages |

**GPU-specific guidance:**
- Use **`by_gpu_id`** to check for systematic card-position effects (e.g., all card-0s behaving differently)
- Use **`by_gpu`** to catch individual GPU outliers — a single underperforming GPU is masked by node-level averages and invisible at the card-slot grouping
- High CV in utilization may indicate load imbalance, but may also reflect intentionally heterogeneous workloads (e.g., data-parallel workers with unequal partition sizes). Do not assume imbalance is a problem without further evidence
- VRAM near 100% = high memory usage (may or may not indicate pressure — some workloads intentionally fill VRAM)
- Non-uniform VRAM across `by_gpu_id` = different GPUs may be doing different work; investigate before labeling as imbalance

**Network/host guidance:**
- Network and host gauges appear in `by_node`; compare per-node uniformity (CV) — low CV with all nodes equally affected points to systemic causes (topology, congestion); high CV points to node-specific issues
- If a counter total (e.g., network bytes) differs between jobs but `by_node` CV is low, the difference is systemic rather than localized to a few nodes

#### Gate check before proceeding

Before moving to iteration analysis or time-series analysis, verify:
- [ ] For every metric that shows a significant anomaly or cross-job difference in the global summary, have you examined the `variance` groupings (`by_node`, `by_gpu_id`, `by_gpu`) to determine whether the issue is systemic or localized?
- [ ] If a metric is uniform in the global summary but you expect variance, have you lowered `--cv-threshold` to confirm it is genuinely uniform rather than below the default gate?
- [ ] If GPU utilization differs, have you checked `by_node` to see whether all nodes are equally affected or if there are outliers?

If the answer to any of these is no, go back and analyze the relevant variance data before proceeding.

### Iteration-Level Analysis

Some workloads have repetitive phases that produce visible idle gaps in the averaged GPU utilization signal. The `iterations` subcommand attempts to detect these boundaries automatically. **However, iteration detection is not always meaningful** — it depends on the workload having a clear, repetitive structure visible in the GPU-averaged utilization signal. Only include iteration analysis in your report if the results are conclusive and consistent.

```bash
# Detect iterations and compute per-iteration stats
omnistat-inspect --tsdb-url $TSDB_URL --cache-dir $SCRATCH/cache job JOBID iterations

# With custom thresholds
omnistat-inspect --tsdb-url $TSDB_URL --cache-dir $SCRATCH/cache job JOBID iterations \
  --low-threshold 15 --high-threshold 75 --min-idle-seconds 20 --min-iteration-seconds 45
```

The `iterations` subcommand:
1. **Identifies iteration boundaries** from averaged GPU utilization — finds sustained idle periods (below `--low-threshold` for at least `--min-idle-seconds`) that separate iterations
2. **Computes per-iteration duration** — often the single most informative metric for detecting performance degradation
3. **Computes the utilization integral** — total GPU-%-seconds per iteration, measuring actual GPU compute work delivered independent of how long it took:
   - If utilization integral is constant across iterations but duration varies → the GPUs do the same work but something else (communication, I/O) takes longer
   - If utilization integral varies → the GPUs are doing different amounts of work per iteration
4. **Counts idle dips** — transitions from high utilization (>`--high-threshold`) to low utilization (<`--low-threshold`) within an iteration
5. **Computes time in utilization bands** — percentage of iteration spent below 20%, below 50%, above 80%, characterizing the balance between compute and communication phases

The `--min-idle-seconds` parameter prevents brief utilization dips within an iteration from being misidentified as iteration boundaries. The `--min-iteration-seconds` parameter filters out spurious short segments at job start/end.

#### When to Report Iteration Results

**Include** iteration analysis when:
- Iterations have consistent durations (low coefficient of variation)
- The number of detected iterations matches what you'd expect from the job's structure
- Per-iteration metrics tell a clear story (e.g., steady-state behavior, or a clear trend)

**Omit or flag as inconclusive** when:
- The detector finds an unexpected or irregular number of iterations
- Iteration durations vary wildly with no clear pattern
- The averaged signal doesn't show clean idle separations (the workload may not be iteration-based)
- Results are more confusing than informative

#### Validating with Per-GPU Analysis

The default iteration detection uses the **averaged** GPU utilization signal — the mean across all GPUs at each time step. This implicitly assumes the workload is roughly homogeneous across GPUs. If the workload is heterogeneous (different GPUs doing different things), the averaged signal may produce misleading iteration boundaries.

To validate, sample a few individual GPUs and compare their iteration structure to the global result:

```bash
# Iteration detection on a single GPU (node + card)
omnistat-inspect --tsdb-url $TSDB_URL --cache-dir $SCRATCH/cache job JOBID query \
  --promql 'rocm_utilization_percentage{instance="HOSTNAME",card="0"} * on (instance) group_left() (max by (instance) (rmsjob_info{$job,$jobstep}))' \
  > $SCRATCH/single_gpu_util.json
```

If individual GPUs show a different iteration pattern than the global average, the workload is likely heterogeneous and the global iteration analysis should not be reported as definitive. Instead, note the heterogeneity as an observation.

**Key insight:** Iteration duration is often a more reliable indicator of performance than mean utilization. Two jobs can have different mean utilization (due to different amounts of idle time) but identical peak performance and compute work — the difference is entirely in how long the communication/idle phases last.

### Annotation-Based Analysis

When `job info` reports annotations (`rmsjob_annotations` markers), analyze each annotated region separately. Annotations mark application phases (e.g., "training", "validation", "checkpoint") or benchmark stages, and different regions often have very different GPU behavior — job-level statistics average over them and can be misleading.

**Discovering annotation time ranges:** Use `timestamp()` to find when each annotation marker was active:

```promql
timestamp(count by (marker) (rmsjob_annotations{$job} > 0))
```

This returns a time series per marker. The first and last timestamps define the region where that annotation was active. Use the `query` subcommand:

```bash
omnistat-inspect --tsdb-url $TSDB_URL --cache-dir $SCRATCH/cache job JOBID query \
  --promql 'timestamp(count by (marker) (rmsjob_annotations{$job} > 0))' \
  > $SCRATCH/annotation_timestamps.json
```

**Per-annotation metrics:** To compute a metric scoped to a specific annotation, join through `rmsjob_info` and `rmsjob_annotations` to propagate the `marker` label. For example, average GPU utilization per annotation region:

```promql
avg by (marker) (
  avg by (instance) (rocm_utilization_percentage)
  * on (instance) group_left(jobid,marker)
    rmsjob_info{$job}
  * on (jobid) group_left(marker)
    count by (jobid,marker) (rmsjob_annotations{$job} > 0)
)
```

This pattern works with any per-node or per-GPU metric. Replace `avg by (instance) (rocm_utilization_percentage)` with the metric of interest.

**What to look for:**
- Do all annotated regions have similar utilization, or do some phases show dramatically different behavior?
- Is FOM concentrated in specific regions?
- Do idle periods between annotations explain overall low utilization?

### Time Series Analysis

For metrics or GPUs that show anomalies in the statistical analysis, fetch the raw time series.

```bash
# Export time series to file (avoids flooding context with large data)
omnistat-inspect --tsdb-url $TSDB_URL --cache-dir $SCRATCH/cache job JOBID timeseries --metric rocm_utilization_percentage > $SCRATCH/util_timeseries.json

# Filter to a specific node or GPU
omnistat-inspect --tsdb-url $TSDB_URL --cache-dir $SCRATCH/cache job JOBID timeseries --metric rocm_utilization_percentage --node hostname1 --card 0 > $SCRATCH/node1_card0.json

# Filter by any other label with --label KEY=VALUE (repeatable). Value may be a
# regex (contains | or .*). Works on both TSDB and CSV. For example, export a
# single hardware counter's raw series by its `name` label:
omnistat-inspect --tsdb-url $TSDB_URL --cache-dir $SCRATCH/cache job JOBID timeseries \
  --metric omnistat_hardware_counter --label name=SQ_INSTS_VALU_FMA_F32 > $SCRATCH/fma_f32.json
omnistat-inspect --tsdb-url $TSDB_URL --cache-dir $SCRATCH/cache job JOBID timeseries \
  --metric omnistat_hardware_counter --label 'name=FETCH_SIZE|WRITE_SIZE' > $SCRATCH/hbm_raw.json
```

`timeseries` exports **raw** series (no aggregation, no rate). **Derived metrics
over time** — FLOPS, HBM bandwidth, and L1/L2 cache hit rate — are computed via the
`query` subcommand using the ready-to-run PromQL documented in the architecture
profile (`gpus/mi250x.md`, `gpus/mi300x.md`); the formulas are arch-specific (e.g.
MI300X includes F8/F6F4 matrix precisions, MI250X does not). This derived-over-time
path is **TSDB-only** (CSV exports have no PromQL engine).

For ad-hoc investigation, use the `query` subcommand with raw PromQL:

```bash
# Custom aggregation -- average utilization across all GPUs over time
omnistat-inspect --tsdb-url $TSDB_URL --cache-dir $SCRATCH/cache job JOBID query \
  --promql 'avg(rocm_utilization_percentage * on (instance) group_left() (max by (instance) (rmsjob_info{$job,$jobstep})))' \
  > $SCRATCH/avg_util.json

# Max temperature per node over time
omnistat-inspect --tsdb-url $TSDB_URL --cache-dir $SCRATCH/cache job JOBID query \
  --promql 'max by (instance) (rocm_temperature_celsius * on (instance) group_left() (max by (instance) (rmsjob_info{$job,$jobstep})))' \
  > $SCRATCH/max_temp_per_node.json
```

### Cross-Metric Reasoning

Move beyond simple correlation to form and test hypotheses about job behavior.

#### Causal Direction

When two metrics are correlated, always ask: **which is cause and which is consequence?** Or are both caused by a third factor?

Template for every correlation:
1. "X is low and Y is low. Does low X cause low Y?"
2. "Or does low Y cause low X?"
3. "Or does some Z cause both low X and low Y?"

**Example from practice:** An investigation found lower network throughput correlated with longer HPL iterations. The initial conclusion — "network bandwidth is the bottleneck" — was challenged: lower throughput could equally be a *consequence* of MPI desynchronization (nodes not ready to receive, so effective throughput drops) rather than a *cause* (fabric unable to deliver bandwidth). The telemetry alone could not distinguish the two. This distinction matters because the remediation is completely different.

**Confidence calibration:** Your reported confidence must match the strength of evidence:
- **High confidence** requires: the data unambiguously points to a single cause, alternative hypotheses have been tested and ruled out, and the finding has been verified at fine resolution
- **Moderate confidence** is appropriate when: the data is consistent with a hypothesis but the causal direction is ambiguous, or the analysis has not been performed at all available granularities
- **Low confidence** is appropriate when: multiple hypotheses are equally consistent with the data

When causal direction is ambiguous — as it often is with correlated metrics like network throughput and GPU idle time — the report **must** present the alternative hypotheses explicitly rather than asserting one as the root cause. Stating "network degradation caused longer iterations" when the data equally supports "something caused MPI desynchronization, which manifests as both lower throughput and longer iterations" is overconfident and potentially misleading.

#### Consequence Chains

Multiple metrics moving together often indicate a single root cause propagating through a chain of effects, not multiple independent problems:

```
More idle time → lower mean utilization → lower mean power → lower mean clocks → lower mean temperature
```

This looks like four separate problems (utilization, power, clocks, temperature) but is actually one (more idle time). **Before listing multiple degraded metrics as separate findings, check whether they are all consequences of a single upstream cause.**

Indicators of a consequence chain:
- All metrics move in the same direction
- The magnitudes are proportional (e.g., 10% less utilization → ~10% less power)
- One metric logically depends on another (GPUs clock down when idle → lower power is a physical consequence, not an independent problem)

Many GPU metrics are physically linked this way: when GPUs idle (waiting for MPI, I/O, or data), utilization drops → GPU clocks reduce (DVFS) → power drops → temperature drops. Mean power, clock speed, and temperature are all **consequences** of utilization, not independent indicators. **When comparing jobs**, do not count lower mean power, lower mean clocks, and lower mean temperature as separate problems if utilization is also lower — they are all effects of the same cause (more idle time).

#### Correlation Patterns

Use these patterns as starting hypotheses, but always verify the causal direction:

| Observation | Possible Interpretation | What to check |
|---|---|---|
| Same peak performance, longer iterations | Communication/I/O bottleneck | Utilization integral (should be constant), network throughput during idle phases |
| Lower peak performance, same iteration duration | GPU compute degradation | Temperature (throttling?), clock speeds, RAS errors |
| High utilization + low FOM | Inefficient compute | Hardware counters if available, memory bandwidth |
| Low utilization + normal power | Memory-bound workload | VRAM usage, HBM bandwidth counters |
| All nodes equally degraded (low CV) | Systemic issue (topology, config) | Node placement, runtime configuration |
| One or few outlier nodes (high CV) | Node-specific issue (hardware, OS) | RAS errors, temperature, per-node stats |

Use the `query` subcommand or `timeseries` exports to examine metrics side-by-side during the same time windows.

## Comparative Analysis Across Jobs

When investigating performance differences between jobs (e.g., healthy vs degraded), single-job analysis is insufficient. You need structured cross-job comparison.

### When to Use Comparative Analysis

- A job is reported as degraded relative to a known baseline
- Multiple jobs run the same workload but achieve different FOM
- You need to determine whether a job's behavior is normal or anomalous

### Establishing a Baseline

1. Identify one or more **healthy reference jobs** running the same workload on the same system
2. Analyze the reference job first (all steps above) to understand normal behavior
3. Use the reference job's statistics as the baseline for comparison

**Critical: Compare at the finest available resolution.** Coarse-step comparisons can be misleading — a job that appears 23% degraded at 60s resolution may have identical peak performance at 5s resolution, with the difference being entirely in iteration duration.

### Systematic Elimination

When comparing healthy and degraded jobs, systematically check each potential cause and either confirm or rule it out:

| Check | What to compare | Rules out |
|-------|----------------|-----------|
| **Peak GPU performance** | Peak FOM or peak utilization at fine resolution | GPU hardware capability |
| **GPU compute work** | Utilization integral per iteration | Workload differences |
| **Iteration duration** | Per-iteration wall-clock time | Communication/I/O overhead |
| **Node balance** | Per-node metric CV and min/max spread | Straggler nodes |
| **GPU-ID balance** | Per-card statistics within nodes | Specific GPU failures |
| **CPU utilization** | Mean active cores, load1, temporal profiles | CPU contention |
| **Memory** | VRAM usage, HBM clocks (MCLK), HBM temperature | Memory issues |
| **Network throughput** | Per-NIC peak rates, sustained rates, total data | Network bottleneck |
| **Hardware health** | RAS errors, thermal throttling, power | Hardware failures |
| **Data collection** | Sampling interval, gaps, monitoring overhead | Measurement artifacts |

For each check, document whether the factor is **the same** (ruled out) or **different** (potential cause). At the end, you should have a short list of factors that actually differ, plus confidence about what does NOT explain the problem.

### Cross-Job Comparison Techniques

When comparing healthy and degraded jobs, apply the variance approach (statistical analysis) to both jobs and compare at each grouping. Key techniques for network and other metrics:

**Per-node comparison:** Compare the `by_node` variance for network metrics across both jobs. If all nodes show proportionally lower throughput in the degraded job, the issue is systemic (topology, congestion). If only specific nodes are degraded, it's localized. Use `timeseries --metric <name> --node <host>` to pull the raw series for any node that stands out, and `db info` to discover available metrics.

**Total data transferred:** Compare cumulative counter deltas (total TX bytes per iteration) rather than just rates. If the same workload transfers the same total data but at lower throughput, the network is delivering the same work more slowly.

**Temporal pattern analysis:** Some workloads have characteristic traffic patterns (e.g., increasing throughput as matrix factorization progresses in HPL). Compare the temporal shape, not just the average — a flattened pattern indicates disrupted communication phases.

**Per-node uniformity:** Compute the coefficient of variation (CV) of per-node metrics for any category. Low CV with all nodes equally affected points to systemic causes. High CV with outlier nodes points to node-specific issues.

## Domain Knowledge

GPU-specific details (power reporting quirks, thermal limits, memory characteristics, RAS error blocks, hardware counter formulas) are documented in the architecture profiles under `gpus/`. The sections below cover concepts that apply universally across GPU architectures.

### RAS Error Interpretation

> **Keep in sync:** the general thresholds below (uncorrectable > 0 → critical

…(truncated)