Fabric Spark Monitoring

Fabric Spark monitoring APIs

Attribute Fabric Spark notebook / pipeline wall-clock time using the Spark monitoring REST surface: Livy session listings plus the embedded Spark History Server (SHS) mirror. Every endpoint here is a GET — read-only diagnostics, no state changes.

All endpoint shapes and field names below were exercised live on 2026-08-17. The surface is preview-era — re-verify response shapes before building anything that depends on their stability.

Auth & base URL

az account get-access-token --resource https://api.fabric.microsoft.com --query accessToken -o tsv

Delegated token. On a Fabric-only tenant (no Azure subscription) the login itself needs az login --allow-no-subscriptions. Base URL is https://api.fabric.microsoft.com/v1; plain curl with a bearer header works from there — no SDK or MCP server required. Full token audience table and 401 debugging: fabric-auth skill.

Listing Livy sessions

GET /workspaces/{wsId}/spark/livySessions?maxResults=N            # workspace-wide
GET /workspaces/{wsId}/notebooks/{itemId}/livySessions            # per-item
GET /workspaces/{wsId}/sparkJobDefinitions/{itemId}/livySessions  #   "
GET /workspaces/{wsId}/lakehouses/{itemId}/livySessions           #   "

Field guide for the session objects:

queuedDuration is pure capacity contention — time spent before the Spark application existed at all. If it dominates, the problem is capacity/queueing, not the notebook.

Drilling into one application

All three item types work here, not just notebooks, and an optional {attemptId} (int) may follow {appId} — omit it and the last attempt is used.

GET .../{notebooks|sparkJobDefinitions|lakehouses}/{itemId}/livySessions/{livyId}/applications/{appId}[/{attemptId}]
GET .../applications/{appId}/jobs        # SHS jobs array
GET .../applications/{appId}/jobs/{id}   # one job
GET .../applications/{appId}/sql/{id}?details=false

The jobs array mirrors the Spark History Server: jobId, name, description, submissionTime, completionTime, status, stageIds, and the num*Tasks / num*Stages counters. The description embeds the originating notebook statement — that is how jobs map back to cells.

/jobs is not the only sub-resource. These are the open-source Spark History Server APIs — same structure, query parameters and contract as Spark's own monitoring REST API — so /stages, /tasks, /executors, /storage, /sql and /logs are all served. (Doc-verified 2026-08-30, not exercised.)

404 trap: exactly two OSS endpoints are withheld — /applications (list all) and /version. That is why .../livySessions/{livyId}/applications with no {appId} returns 404: there is no "list applications" form, by design. Get sparkApplicationId from a session listing instead.

The attribution recipe

Decompose notebook wall-clock into five phases:

1. Queued — queuedDuration from the session listing.
2. Livy → app — Livy startDateTime to the app attempt's start.
3. Boot + non-Spark cells — app start to first Spark job submission: cluster boot plus cells that never touch Spark (imports, pyodbc control reads, pure-Python setup).
4. Real work — first to last Spark job. Sum job durations and compute inter-job gaps; gaps are driver-side / non-Spark time between cells. Overlapping submission windows prove intra-notebook parallelism.
5. Teardown — last job completion to app end (snapshot/teardown).

Worked example (live run, 2026-08-17): a 6m48s notebook decomposed to 101s queue / 31s Livy→app / 107s boot + pre-cells / 73s Spark jobs (with 17s of dead gaps) / ~75s teardown — i.e. only ~18% of wall-clock was Spark work.

Gap analysis over the jobs array:

from datetime import datetime

def ts(s):  # SHS timestamps end in literal "GMT" — strip it first
    return datetime.strptime(s.replace("GMT", ""), "%Y-%m-%dT%H:%M:%S.%f")

jobs.sort(key=lambda j: ts(j["submissionTime"]))
prev_end, gaps = None, []
for j in jobs:
    start, end = ts(j["submissionTime"]), ts(j["completionTime"])
    if prev_end is not None and (gap := (start - prev_end).total_seconds()) > 1:
        gaps.append((j["jobId"], round(gap, 1)))
    prev_end = end if prev_end is None else max(prev_end, end)

Logs and resource usage

Two more sibling routes hang off .../applications/{appId}/:

• /logs?type=livy|driver|rollingdriver|executor — Livy, driver (single and rolling) and per-executor log metadata and content.
• /resourceUsage — coreEfficiency, idleTime and a per-timestamp core / executor / job timeline. It reports directly what the recipe above infers by subtraction, so reach for it before hand-rolling gap analysis.

Route table, query parameters, response fields and the traps (the literal applications/none Livy-log form, capacityExceeded emptying the payload, driver-log offset being ignored once the application stops) are in references/REFERENCE.md. Doc-verified 2026-08-30, not exercised.

High-concurrency session semantics

The reuse verdict lives in the pipeline Notebook activity's run output: highConcurrencyModeStatus.sessionSource — created (paid the full startup cost) vs reused (attached to a live session). sessionTag groups runs onto the same session.

Reuse eliminates phases 1–3 entirely — for small-data runs those are the dominant cost, so reuse is usually the single biggest lever.

Reuse conditions (all must hold): same session tag, same submitting user, same default lakehouse + compute configuration, target session still alive, and ≤ 5 notebooks already attached to the session.

Gotchas

• SHS timestamps end in literal GMT (e.g. 2026-08-17T14:03:21.117GMT) — strip it before strptime; %Z will not parse it reliably.
• Windows path split: curl under Git Bash writing to /tmp produces files Windows-side Python can't see (Git Bash's /tmp is not C:\tmp). Write API responses to a repo-relative or scratchpad path instead.
• Preview-era surface — these monitoring APIs predate a stable GA contract; verify response shapes before relying on them.

See also

• fabric-warehouse-monitoring skill — Warehouse-side counterpart (queryinsights, query labels, DMVs)
• fabric-rest-api skill — generic auth, LRO, pagination, item-ID patterns
• fabric-spark skill — notebook authoring, session configuration, high-concurrency setup
• fabric-auth skill — token audiences and 401 debugging