Job Analysis

Job Analysis

Analyze GPU telemetry data collected by Omnistat for HPC/AI workloads. This skill guides you through a top-down, hypothesis-driven analysis of job performance, driven primarily by the omnistat-inspect CLI tool.

Target audience: HPC engineers, AI/ML researchers, system administrators investigating job performance, GPU health, and resource utilization.

What the analysis produces: A structured report identifying performance bottlenecks, hardware issues, resource utilization patterns, and anomalies -- with all findings backed by data.

When to use this vs job-report. Use job-analysis when you need to understand why a job behaved as it did — diagnosing bottlenecks, throttling, stragglers, or regressions, or comparing a degraded job against a healthy baseline. It is hypothesis-driven and iterative. If you only need a quick factual snapshot of what a job did (stats, energy, health, data quality) without investigation, use job-report instead. A common pattern is to run job-report first, then reach for job-analysis when something looks off.

Tooling: omnistat-inspect

This skill is built entirely around omnistat-inspect, the consolidated analysis CLI. It is your single entry point for every phase:

• Baseline characterization — omnistat-inspect --tsdb-url $TSDB_URL job JOBID report produces the structured report card (overview, stats, variance, data-collection, health). Start every analysis here: it is the fastest way to understand scale, runtime, utilization, variance, and health in a single call.
• Deep-dive subcommands — job JOBID info (metadata/topology), stats (gauges, counters, hardware counters, and per-node/per-GPU variance), health (data-collection coverage and health checks), iterations (iteration boundaries and per-iteration stats), query (arbitrary PromQL), and timeseries (raw series export).
• Data-source inspection — omnistat-inspect --tsdb-url $TSDB_URL db info lists the jobs and metrics available in the backend (no job context required).

For anything not covered by a subcommand, drop to raw PromQL via query (TSDB) or curl against the TSDB HTTP API.

Job-context flexibility

Every omnistat-inspect job JOBID invocation resolves the job's time window in one of two ways:

• Discovery (default): omits --start/--end; the tool scans the database to discover the job's time range and topology. Add --cache-dir DIR to persist the discovery snapshot and per-section results so repeat calls are cheap (no re-scan, no re-query).
• Direct window: pass both --start ISO8601 and --end ISO8601 (optionally --interval SECONDS) to skip discovery entirely and analyze an exact window — useful for zooming into a single phase or iteration you found earlier.

# Discovery + cache (cheap repeat calls)
omnistat-inspect --tsdb-url $TSDB_URL --cache-dir $SCRATCH/cache job JOBID report

# Direct window (no discovery scan)
omnistat-inspect --tsdb-url $TSDB_URL job JOBID \
  --start 2026-01-01T12:00:00Z --end 2026-01-01T12:10:00Z report

Cache-dir reuse for late-pipeline query / timeseries

The query and timeseries subcommands are typically run late in the analysis, well after discovery. Always pass the same --cache-dir you used for the initial report/info call so they rehydrate the cached discovery snapshot instead of re-scanning:

# Early: discovery runs once and is cached (time range + sampling interval)
omnistat-inspect --tsdb-url $TSDB_URL --cache-dir $SCRATCH/cache job JOBID info

# Later: query/timeseries reuse the snapshot — no re-scan, correct default step
omnistat-inspect --tsdb-url $TSDB_URL --cache-dir $SCRATCH/cache job JOBID \
  query --promql 'avg(rocm_utilization_percentage{$job, $jobstep})'

Because the rehydrated snapshot restores the discovered sampling interval, the default query step is the sampling interval (max(sampling_interval, 1s)) — exactly what you want for full-resolution queries. Notes:

• Do not reach for --start/--end just to "scope to the discovered window" — passing them skips discovery and drops the sampling interval, so the default step silently degrades to 1s unless you also add --interval. Use --start/--end only when you genuinely want a narrower sub-window, and pair them with --interval to keep the step correct.
• For a deliberately coarser step on a single query (e.g. an overview of a long job), pass --step SECONDS directly; it overrides the default for that call only. (timeseries has no --step; control its resolution via the cached interval or the global --interval.)

Prerequisites

1. Data source — one of:
   • VictoriaMetrics running with the Omnistat database loaded (use the open-database skill if needed), OR
   • CSV exports from omnistat-query --export (no TSDB required)
2. Python virtual environment activated with omnistat installed (pip install ".[query]" from the omnistat repo root) — this provides omnistat-inspect. Confirm with which omnistat-inspect.
3. Job ID(s) to analyze (discover available jobs with omnistat-inspect --tsdb-url $TSDB_URL db info or omnistat-inspect --csv-dir /path/to/exports db info)

Setup

Before starting analysis, set up the working environment:

TSDB Mode (default)

# 1. Create a cache directory for this analysis session (cheap repeat calls)
SCRATCH=$(mktemp -d /tmp/omnistat-inspect-XXXXXX)
echo "Cache directory: $SCRATCH/cache"

# 2. Set the TSDB URL (VictoriaMetrics or Prometheus)
TSDB_URL="http://localhost:8428"

# 3. Verify connectivity and discover available jobs
omnistat-inspect --tsdb-url $TSDB_URL db info

CSV Mode

Use CSV mode when you have CSV exports from omnistat-query --export and no running TSDB. All subcommands except job query work in CSV mode — CSV mode uses whatever metrics were exported, so if a metric wasn't included in the export, it won't be available for analysis.

# 1. Create a cache directory for this analysis session
SCRATCH=$(mktemp -d /tmp/omnistat-inspect-XXXXXX)
echo "Cache directory: $SCRATCH/cache"

# 2. Set the CSV directory path
CSV_DIR="/path/to/csv/exports"

# 3. Discover available data
omnistat-inspect --csv-dir $CSV_DIR db info

# 4. Run analysis (same subcommands as TSDB mode)
omnistat-inspect --csv-dir $CSV_DIR --cache-dir $SCRATCH/cache job JOBID info
omnistat-inspect --csv-dir $CSV_DIR --cache-dir $SCRATCH/cache job JOBID stats
omnistat-inspect --csv-dir $CSV_DIR --cache-dir $SCRATCH/cache job JOBID health

Note: The job query subcommand (arbitrary PromQL) is not available in CSV mode — it requires a TSDB backend.

The db info subcommand verifies database connectivity and reports all available jobs with their time ranges, node counts, users, and partitions, plus the full list of available metrics. Use this output to select a job ID and confirm you are looking at the right database.

Analysis Workflow

Follow this top-down, hypothesis-driven workflow. Each phase builds on the previous one. You have freedom to explore and investigate -- this is a methodology guide, not a rigid script.

Epistemic Discipline

Do not assume what the workload is. Unless the user tells you the application name, or annotations/metadata explicitly identify it, treat the workload as unknown. Describe what the telemetry shows (e.g., "the GPUs spend ~40% of wall-clock idle between compute phases, each phase ~90s long") rather than what you think it means (e.g., "this is a training workload doing forward/backward passes"). Note: GPU utilization naturally sits near 0% or near 100%, so simply calling it "bimodal" is not an insight — quantify the idle fraction or phase structure instead. If you need to speculate, label it clearly as a hypothesis.

Do not assume the workload is homogeneous. A single HPC job may run different tasks on different nodes or GPUs. Some nodes may run data loading, others may run compute, others may handle communication. VRAM differences across GPUs, utilization variance across nodes, or non-uniform network traffic are signals of heterogeneity, not necessarily problems. Before reporting "imbalance" as a finding, consider whether the workload is intentionally heterogeneous.

Report what you observe, not what you expect. If a metric looks unusual, describe the observation and its magnitude. Do not assume it is a problem unless you have evidence of impact (e.g., on runtime, throughput, or health). An observation like "5% of GPUs use 10x more VRAM than the rest" is a fact; "there is a memory imbalance problem" is an interpretation that may be wrong.

Job Discovery and Characterization

The first step of every analysis is the one-shot report. It is the factual baseline the rest of the workflow builds on — a single call that returns the job overview, the full stats block (gauges, counters, hardware counters, variance), and the health block (data-collection coverage + hardware health). Save it and reuse its embedded blocks; the downstream sections below consume this output rather than re-fetching the same data.

# First step — factual baseline. Embeds overview + stats + health in one JSON.
omnistat-inspect --tsdb-url $TSDB_URL --cache-dir $SCRATCH/cache job JOBID report > $SCRATCH/report_JOBID.json

# List all metrics available in the data source, plus all jobs and time ranges
omnistat-inspect --tsdb-url $TSDB_URL db info

The report JSON has top-level keys overview, stats, and health. Everything the "Data Collection and Hardware Health Validation" and "Statistical Analysis" sections need is already in this one document — you only issue additional stats / health / info calls to drill down (finer grouping), refresh (after --interval changes), or when you deliberately skipped the baseline report. job info on its own returns just the overview subset if you ever need it in isolation.

Key information to extract:

• Runtime: How long did the job run?
• Scale: How many nodes and GPUs?
• Sampling interval: What time resolution is available?
• Available metrics: Which collectors were active? (GPU, host, network, RAS, xGMI, rocprofiler) — db info lists every metric present in the backend.
• Annotations: rmsjob_annotations markers (e.g., application phases, benchmark identifiers)
• Figure of Merit: omnistat_fom values (e.g., GFLOPS achieved)

The job info subcommand automatically includes annotations and figure_of_merit when the corresponding metrics are present in the database.

The job info subcommand reports the discovered sampling interval (auto-detected from the omnistat_info metric's interval_secs label). The sampling interval is also used internally by stats, health, and iterations to auto-compute the finest safe query step — you do not need to pass --interval to these subcommands.

GPU Architecture Detection

After discovering the job, identify the GPU architecture from the available metrics and load the corresponding architecture profile for GPU-specific domain knowledge (power reporting quirks, thermal limits, memory characteristics, RAS error blocks, hardware counter formulas).

Architecture profiles are located in skills/job-analysis/gpus/. Read the matching profile before proceeding to data-collection and health validation.

Detection: Use overview.gpu_type from the job info output and apply the same substring rules as the job-report skill's "GPU Architecture Handling" table (MI250 or MI200 (MCM) → MI250X; MI300 → MI300X). When gpu_type is a list, apply the rule to each element.

The architecture profile contains critical information for correct interpretation of the data (e.g., which GPU cards report power, thermal throttling thresholds, RAS error block meanings).

Resolution Sensitivity

Step resolution significantly affects observed statistics. Coarse steps (e.g., 60s) average over intervals, smearing peaks and troughs together. This can be seriously misleading:

• Peak metrics are underestimated at coarse resolution (e.g., peak FOM at 60s may be 10-25% lower than at 5s)
• Mean metrics are mostly unaffected by resolution (averaging preserves the mean)
• Iteration boundaries blur at coarse resolution, making it impossible to distinguish per-iteration behavior

Always verify critical findings at the finest feasible resolution. The finest meaningful resolution is the sampling interval reported by job info (from omnistat_info's interval_secs label) — querying at a finer step than this adds no real data.

Step Selection

The stats, health, and iterations subcommands auto-compute the finest safe query step. The step is max(sampling_interval, runtime / 90000) — never finer than the actual data, never exceeding VictoriaMetrics' search.maxPointsPerTimeseries limit (90,000). There is no arbitrary floor: sub-second sampling intervals are preserved for short jobs where VM limits allow it.

--interval is a flag on the job group and must be placed before the subcommand (e.g. job JOBID --interval N stats), not after it. The iterations subcommand ignores --interval — it always uses an auto-computed step. For stats and health it refines the time range only, while the query step stays auto-computed.

For timeseries and query, the default step is the discovered sampling interval (max(sampling_interval, 1s)) when you reuse the cached discovery snapshot via --cache-dir — full resolution with no extra flags. For a coarser overview on a long job, query accepts an explicit --step SECONDS; timeseries has no --step, so adjust its resolution via the cached interval or the global --interval.

When the auto-computed step is much coarser than the sampling interval (which happens on very long jobs), state the resolution gap explicitly in the report and note which findings may be affected (especially peaks and percentiles).

Critical rule for peak metrics: If peak FOM, peak utilization, or peak throughput appears degraded, always re-verify at the finest feasible step (using query with an explicit --step) before concluding there is a peak performance difference. Apparent peak degradation is frequently an artifact of temporal averaging — the true peaks may be identical across jobs. Do not claim peak performance differs without checking at fine resolution.

Data Collection and Hardware Health Validation

Before analyzing performance, verify that data collection was complete and reliable, and check for hardware issues. This data is already in the baseline report's health block — read it from there; do not re-fetch. Only run health standalone if you skipped the baseline report or need to refresh after changing --interval:

# Standalone health (only if you skipped the baseline report or are refreshing)
omnistat-inspect --tsdb-url $TSDB_URL --cache-dir $SCRATCH/cache job JOBID health

The health block covers both data-collection coverage (completeness, timing stagger, gaps) and hardware health (RAS errors, thermals, power).

Data-collection coverage (part of health)

Review the coverage portion of the health report for:

• Missing nodes: expected_nodes vs reporting_nodes — any gap means some nodes never reported
• Activation stagger: activation_stagger_seconds — how long it took for all nodes to start reporting. A spread >5% of total job duration is significant and means early-job statistics are skewed by partial participation
• Deactivation stagger: deactivation_stagger_seconds — same for shutdown. Large spread means late-job statistics are unreliable
• Sampling gaps: nodes_with_gaps and total_gaps — gaps are reported only as counts, not per-gap timing. To localize them (e.g., distinguish a clustered systemic event from distributed per-node issues), drill down with timeseries/query
• Reporting duration: reporting_duration_per_node_seconds (a {mean, min, max} object) — nodes with significantly shorter reporting durations may have crashed or been evicted mid-job

Hardware health (health)

Review the health report for:

• RAS errors: Any hardware errors during the job
• Thermal issues: GPUs running hot
• Power anomalies: Unexpected zero-power readings
• Push health: Whether monitoring push duration exceeded the push interval (indicates monitoring overhead)

If critical issues are found, note them -- they may explain performance anomalies found later.

Statistical Analysis

Follow these steps in order. Do not skip steps or move to iteration analysis until all steps are complete.

Step 1: Read the baseline stats

The job's stats are already in the baseline report (report_JOBID.json → stats): global gauge/counter summaries, hardware counters, and per-node / per-GPU variance, all in one block — no --category or --level flags exist or are needed. Read them from the baseline; only run stats standalone if you skipped the report or are refreshing after an --interval change:

# Standalone stats (only if you skipped the baseline report or are refreshing)
omnistat-inspect --tsdb-url $TSDB_URL --cache-dir $SCRATCH/cache job JOBID stats > $SCRATCH/stats_JOBID.json

The stats block has keys gauges, counters, hardware_counters, and variance. Counter metrics (cumulative values like bytes transferred, energy consumed) are automatically detected and produce delta-based totals. Gauge metrics produce mean/min/max/cv/percentiles. The cv (coefficient of variation) field measures relative dispersion — high CV indicates non-uniform distribution across GPUs or nodes.

Step 2: Identify anomalous metrics

Review the gauge and counter stats. For each metric, check for:

• High cv (uneven distribution across nodes/GPUs)
• Unexpected values (rates, totals, or distributions that differ from expectation)
• Large gaps between percentiles (for GPU utilization specifically, a near-0/near-100 split is the expected norm — not a finding; quantify idle time or phase structure rather than labeling it "bimodal")

In comparative analysis: compare each metric between the healthy and degraded jobs. Identify which show significant differences (>10% in rates or totals, >5 percentage points in gauge means).

Step 3: Inspect the variance breakdown

The variance block in the same stats output already drills into any metric whose between-node or between-GPU CV exceeds the threshold (default cv_threshold=0.05; override with --cv-threshold). Each entry first collapses every spatial key's whole-job series to one scalar — the reduction field names which one: temporal_mean (time-average of samples; plain gauges), rate (Δtotal ÷ active duration; counter-derived gauges like network RX/TX), ratio (Δa ÷ Δb between two counters; kernel mean dispatch duration = Δduration ÷ Δdispatches), or total (a single reset-aware cumulative-counter delta per key). Counter metrics (COUNTER_LIST: IO, network, vendor energy, xGMI) now participate in stats.variance with reduction: "total", grouped by source like gauges — every counter gets a by_node entry; GPU-source counters (xGMI) also get by_gpu/by_gpu_id. It then carries the between-key cv (dispersion across those per-key reduced values), min/max (the lowest/highest per-key reduced value and the key that owns it — extremes of per-key reductions, not absolute sample minima/maxima, which live in stats.gauges[].min/max), and either an all map listing every key (when n ≤ 16) or percentiles over the per-key values (when n > 16; --verbose adds all too). The three groupings:

• by_node — one reduced value per node ({instance}); straggler nodes, systemic vs. node-local effects
• by_gpu_id — one per card slot ({card}); card-position effects
• by_gpu — one per (node, card) ({instance, card}); individual GPU stragglers

Use --verbose to force full per-entity arrays even for large jobs. To raise sensitivity, lower --cv-threshold:

# More sensitive variance drill-down: run for BOTH jobs
omnistat-inspect --tsdb-url $TSDB_URL --cache-dir $SCRATCH/cache job JOBID stats --cv-threshold 0.02 --verbose > $SCRATCH/stats_verbose_JOBID.json

The step is auto-computed to stay within maxPointsPerTimeseries limits, so queries should not fail due to point limits. If a query does fail, the --interval flag can be used as an override to force a coarser step.

The variance breakdown answers critical questions that the global summary cannot:

• Is the anomaly systemic (all nodes/GPUs equally affected) or localized?
• Is there a straggler node or a systematic card-position effect?

In comparative analysis: examine the variance block for both the healthy and degraded jobs so you can compare at each grouping.

The variance block exposes three groupings, from coarse to fine:

GPU-specific guidance:

• Use by_gpu_id to check for systematic card-position effects (e.g., all card-0s behaving differently)
• Use by_gpu to catch individual GPU outliers — a single underperforming GPU is masked by node-level averages and invisible at the card-slot grouping
• High CV in utilization may indicate load imbalance, but may also reflect intentionally heterogeneous workloads (e.g., data-parallel workers with unequal partition sizes). Do not assume imbalance is a problem without further evidence
• VRAM near 100% = high memory usage (may or may not indicate pressure — some workloads intentionally fill VRAM)
• Non-uniform VRAM across by_gpu_id = different GPUs may be doing different work; investigate before labeling as imbalance

Network/host guidance:

• Network and host gauges appear in by_node; compare per-node uniformity (CV) — low CV with all nodes equally affected points to systemic causes (topology, congestion); high CV points to node-specific issues
• If a counter total (e.g., network bytes) differs between jobs but by_node CV is low, the difference is systemic rather than localized to a few nodes

Gate check before proceeding

Before moving to iteration analysis or time-series analysis, verify:

• For every metric that shows a significant anomaly or cross-job difference in the global summary, have you examined the variance groupings (by_node, by_gpu_id, by_gpu) to determine whether the issue is systemic or localized?
• If a metric is uniform in the global summary but you expect variance, have you lowered --cv-threshold to confirm it is genuinely uniform rather than below the default gate?
• If GPU utilization differs, have you checked by_node to see whether all nodes are equally affected or if there are outliers?

If the answer to any of these is no, go back and analyze the relevant variance data before proceeding.

Iteration-Level Analysis

Some workloads have repetitive phases that produce visible idle gaps in the averaged GPU utilization signal. The iterations subcommand attempts to detect these boundaries automatically. However, iteration detection is not always meaningful — it depends on the workload having a clear, repetitive structure visible in the GPU-averaged utilization signal. Only include iteration analysis in your report if the results are conclusive and consistent.

# Detect iterations and compute per-iteration stats
omnistat-inspect --tsdb-url $TSDB_URL --cache-dir $SCRATCH/cache job JOBID iterations

# With custom thresholds
omnistat-inspect --tsdb-url $TSDB_URL --cache-dir $SCRATCH/cache job JOBID iterations \
  --low-threshold 15 --high-threshold 75 --min-idle-seconds 20 --min-iteration-seconds 45

The iterations subcommand:

1. Identifies iteration boundaries from averaged GPU utilization — finds sustained idle periods (below --low-threshold for at least --min-idle-seconds) that separate iterations
2. Computes per-iteration duration — often the single most informative metric for detecting performance degradation
3. Computes the utilization integral — total GPU-%-seconds per iteration, measuring actual GPU compute work delivered independent of how long it took:
   • If utilization integral is constant across iterations but duration varies → the GPUs do the same work but something else (communication, I/O) takes longer
   • If utilization integral varies → the GPUs are doing different amounts of work per iteration
4. Counts idle dips — transitions from high utilization (>--high-threshold) to low utilization (<--low-threshold) within an iteration
5. Computes time in utilization bands — percentage of iteration spent below 20%, below 50%, above 80%, characterizing the balance between compute and communication phases

The --min-idle-seconds parameter prevents brief utilization dips within an iteration from being misidentified as iteration boundaries. The --min-iteration-seconds parameter filters out spurious short segments at job start/end.

When to Report Iteration Results

Include iteration analysis when:

• Iterations have consistent durations (low coefficient of variation)
• The number of detected iterations matches what you'd expect from the job's structure
• Per-iteration metrics tell a clear story (e.g., steady-state behavior, or a clear trend)

Omit or flag as inconclusive when:

• The detector finds an unexpected or irregular number of iterations
• Iteration durations vary wildly with no clear pattern
• The averaged signal doesn't show clean idle separations (the workload may not be iteration-based)
• Results are more confusing than informative

Validating with Per-GPU Analysis

The default iteration detection uses the averaged GPU utilization signal — the mean across all GPUs at each time step. This implicitly assumes the workload is roughly homogeneous across GPUs. If the workload is heterogeneous (different GPUs doing different things), the averaged signal may produce misleading iteration boundaries.

To validate, sample a few individual GPUs and compare their iteration structure to the global result:

# Iteration detection on a single GPU (node + card)
omnistat-inspect --tsdb-url $TSDB_URL --cache-dir $SCRATCH/cache job JOBID query \
  --promql 'rocm_utilization_percentage{instance="HOSTNAME",card="0"} * on (instance) group_left() (max by (instance) (rmsjob_info{$job,$jobstep}))' \
  > $SCRATCH/single_gpu_util.json

If individual GPUs show a different iteration pattern than the global average, the workload is likely heterogeneous and the global iteration analysis should not be reported as definitive. Instead, note the heterogeneity as an observation.

Key insight: Iteration duration is often a more reliable indicator of performance than mean utilization. Two jobs can have different mean utilization (due to different amounts of idle time) but identical peak performance and compute work — the difference is entirely in how long the communication/idle phases last.

Annotation-Based Analysis

When job info reports annotations (rmsjob_annotations markers), analyze each annotated region separately. Annotations mark application phases (e.g., "training", "validation", "checkpoint") or benchmark stages, and different regions often have very different GPU behavior — job-level statistics average over them and can be misleading.

Discovering annotation time ranges: Use timestamp() to find when each annotation marker was active:

timestamp(count by (marker) (rmsjob_annotations{$job} > 0))

This returns a time series per marker. The first and last timestamps define the region where that annotation was active. Use the query subcommand:

omnistat-inspect --tsdb-url $TSDB_URL --cache-dir $SCRATCH/cache job JOBID query \
  --promql 'timestamp(count by (marker) (rmsjob_annotations{$job} > 0))' \
  > $SCRATCH/annotation_timestamps.json

Per-annotation metrics: To compute a metric scoped to a specific annotation, join through rmsjob_info and rmsjob_annotations to propagate the marker label. For example, average GPU utilization per annotation region:

avg by (marker) (
  avg by (instance) (rocm_utilization_percentage)
  * on (instance) group_left(jobid,marker)
    rmsjob_info{$job}
  * on (jobid) group_left(marker)
    count by (jobid,marker) (rmsjob_annotations{$job} > 0)
)

This pattern works with any per-node or per-GPU metric. Replace avg by (instance) (rocm_utilization_percentage) with the metric of interest.

What to look for:

• Do all annotated regions have similar utilization, or do some phases show dramatically different behavior?
• Is FOM concentrated in specific regions?
• Do idle periods between annotations explain overall low utilization?

Time Series Analysis

For metrics or GPUs that show anomalies in the statistical analysis, fetch the raw time series.

# Export time series to file (avoids flooding context with large data)
omnistat-inspect --tsdb-url $TSDB_URL --cache-dir $SCRATCH/cache job JOBID timeseries --metric rocm_utilization_percentage > $SCRATCH/util_timeseries.json

# Filter to a specific node or GPU
omnistat-inspect --tsdb-url $TSDB_URL --cache-dir $SCRATCH/cache job JOBID timeseries --metric rocm_utilization_percentage --node hostname1 --card 0 > $SCRATCH/node1_card0.json

# Filter by any other label with --label KEY=VALUE (repeatable). Value may be a
# regex (contains | or .*). Works on both TSDB and CSV. For example, export a
# single hardware counter's raw series by its `name` label:
omnistat-inspect --tsdb-url $TSDB_URL --cache-dir $SCRATCH/cache job JOBID timeseries \
  --metric omnistat_hardware_counter --label name=SQ_INSTS_VALU_FMA_F32 > $SCRATCH/fma_f32.json
omnistat-inspect --tsdb-url $TSDB_URL --cache-dir $SCRATCH/cache job JOBID timeseries \
  --metric omnistat_hardware_counter --label 'name=FETCH_SIZE|WRITE_SIZE' > $SCRATCH/hbm_raw.json

timeseries exports raw series (no aggregation, no rate). Derived metrics over time — FLOPS, HBM bandwidth, and L1/L2 cache hit rate — are computed via the query subcommand using the ready-to-run PromQL documented in the architecture profile (gpus/mi250x.md, gpus/mi300x.md); the formulas are arch-specific (e.g. MI300X includes F8/F6F4 matrix precisions, MI250X does not). This derived-over-time path is TSDB-only (CSV exports have no PromQL engine).

For ad-hoc investigation, use the query subcommand with raw PromQL:

# Custom aggregation -- average utilization across all GPUs over time
omnistat-inspect --tsdb-url $TSDB_URL --cache-dir $SCRATCH/cache job JOBID query \
  --promql 'avg(rocm_utilization_percentage * on (instance) group_left() (max by (instance) (rmsjob_info{$job,$jobstep})))' \
  > $SCRATCH/avg_util.json

# Max temperature per node over time
omnistat-inspect --tsdb-url $TSDB_URL --cache-dir $SCRATCH/cache job JOBID query \
  --promql 'max by (instance) (rocm_temperature_celsius * on (instance) group_left() (max by (instance) (rmsjob_info{$job,$jobstep})))' \
  > $SCRATCH/max_temp_per_node.json

Cross-Metric Reasoning

Move beyond simple correlation to form and test hypotheses about job behavior.

Causal Direction

When two metrics are correlated, always ask: which is cause and which is consequence? Or are both caused by a third factor?

Template for every correlation:

1. "X is low and Y is low. Does low X cause low Y?"
2. "Or does low Y cause low X?"
3. "Or does some Z cause both low X and low Y?"

Example from practice: An investigation found lower network throughput correlated with longer HPL iterations. The initial conclusion — "network bandwidth is the bottleneck" — was challenged: lower throughput could equally be a consequence of MPI desynchronization (nodes not ready to receive, so effective throughput drops) rather than a cause (fabric unable to deliver bandwidth). The telemetry alone could not distinguish the two. This distinction matters because the remediation is completely different.

Confidence calibration: Your reported confidence must match the strength of evidence:

• High confidence requires: the data unambiguously points to a single cause, alternative hypotheses have been tested and ruled out, and the finding has been verified at fine resolution
• Moderate confidence is appropriate when: the data is consistent with a hypothesis but the causal direction is ambiguous, or the analysis has not been performed at all available granularities
• Low confidence is appropriate when: multiple hypotheses are equally consistent with the data

When causal direction is ambiguous — as it often is with correlated metrics like network throughput and GPU idle time — the report must present the alternative hypotheses explicitly rather than asserting one as the root cause. Stating "network degradation caused longer iterations" when the data equally supports "something caused MPI desynchronization, which manifests as both lower throughput and longer iterations" is overconfident and potentially misleading.

Consequence Chains

Multiple metrics moving together often indicate a single root cause propagating through a chain of effects, not multiple independent problems:

More idle time → lower mean utilization → lower mean power → lower mean clocks → lower mean temperature

This looks like four separate problems (utilization, power, clocks, temperature) but is actually one (more idle time). Before listing multiple degraded metrics as separate findings, check whether they are all consequences of a single upstream cause.

Indicators of a consequence chain:

• All metrics move in the same direction
• The magnitudes are proportional (e.g., 10% less utilization → ~10% less power)
• One metric logically depends on another (GPUs clock down when idle → lower power is a physical consequence, not an independent problem)

Many GPU metrics are physically linked this way: when GPUs idle (waiting for MPI, I/O, or data), utilization drops → GPU clocks reduce (DVFS) → power drops → temperature drops. Mean power, clock speed, and temperature are all consequences of utilization, not independent indicators. When comparing jobs, do not count lower mean power, lower mean clocks, and lower mean temperature as separate problems if utilization is also lower — they are all effects of the same cause (more idle time).

Correlation Patterns

Use these patterns as starting hypotheses, but always verify the causal direction:

Use the query subcommand or timeseries exports to examine metrics side-by-side during the same time windows.

Comparative Analysis Across Jobs

When investigating performance differences between jobs (e.g., healthy vs degraded), single-job analysis is insufficient. You need structured cross-job comparison.

When to Use Comparative Analysis

• A job is reported as degraded relative to a known baseline
• Multiple jobs run the same workload but achieve different FOM
• You need to determine whether a job's behavior is normal or anomalous

Establishing a Baseline

1. Identify one or more healthy reference jobs running the same workload on the same system
2. Analyze the reference job first (all steps above) to understand normal behavior
3. Use the reference job's statistics as the baseline for comparison

Critical: Compare at the finest available resolution. Coarse-step comparisons can be misleading — a job that appears 23% degraded at 60s resolution may have identical peak performance at 5s resolution, with the difference being entirely in iteration duration.

Systematic Elimination

When comparing healthy and degraded jobs, systematically check each potential cause and either confirm or rule it out:

For each check, document whether the factor is the same (ruled out) or different (potential cause). At the end, you should have a short list of factors that actually differ, plus confidence about what does NOT explain the problem.

Cross-Job Comparison Techniques

When comparing healthy and degraded jobs, apply the variance approach (statistical analysis) to both jobs and compare at each grouping. Key techniques for network and other metrics:

Per-node comparison: Compare the by_node variance for network metrics across both jobs. If all nodes show proportionally lower throughput in the degraded job, the issue is systemic (topology, congestion). If only specific nodes are degraded, it's localized. Use timeseries --metric <name> --node <host> to pull the raw series for any node that stands out, and db info to discover available metrics.

Total data transferred: Compare cumulative counter deltas (total TX bytes per iteration) rather than just rates. If the same workload transfers the same total data but at lower throughput, the network is delivering the same work more slowly.

Temporal pattern analysis: Some workloads have characteristic traffic patterns (e.g., increasing throughput as matrix factorization progresses in HPL). Compare the temporal shape, not just the average — a flattened pattern indicates disrupted communication phases.

Per-node uniformity: Compute the coefficient of variation (CV) of per-node metrics for any category. Low CV with all nodes equally affected points to systemic causes. High CV with outlier nodes points to node-specific issues.

Domain Knowledge

GPU-specific details (power reporting quirks, thermal limits, memory characteristics, RAS error blocks, hardware counter formulas) are documented in the architecture profiles under gpus/. The sections below cover concepts that apply universally across GPU architectures.

RAS Error Interpretation

Keep in sync: the general thresholds below (uncorrectable > 0 → critical

…(truncated)