Fabric Capacity Operations

Fabric Capacity Operations

1. Overview

Microsoft Fabric capacities are the compute resources that power all Fabric workloads. A capacity is defined by its SKU (F2 through F2048), which determines the number of Capacity Units (CUs) available. Every operation in Fabric — Spark notebook runs, KQL queries, Data Factory pipeline executions, report refreshes, and lakehoue reads — consumes CUs. Understanding how CU consumption works, how throttling is applied, and how to tune workloads to stay within capacity limits is essential for reliable, cost-effective Fabric operations.

SKU Overview:

2. Quick Start

Assess Capacity Health

# Get capacity details and status
curl "https://api.fabric.microsoft.com/v1/capacities/${CAPACITY_ID}" \
  -H "Authorization: Bearer ${TOKEN}"

# List all capacities in tenant (admin)
curl "https://api.powerbi.com/v1.0/myorg/admin/capacities" \
  -H "Authorization: Bearer ${ADMIN_TOKEN}"

Check for Throttling

// In Fabric Capacity Metrics app — check if capacity is being throttled
capacityMetrics
| where TimeGenerated > ago(24h)
| where MetricName == "ThrottlingPercentage"
| summarize MaxThrottling = max(MetricValue) by bin(TimeGenerated, 5m)
| where MaxThrottling > 0
| order by TimeGenerated desc

3. Core Concepts

Capacity Units (CUs) and Smoothing

CU consumption is smoothed over a 24-hour rolling window. Smoothing means that short bursts of high CU consumption are averaged out, allowing workloads to temporarily exceed the SKU's CU limit if the overall rolling average is within budget.

Smoothing tiers:

Interactive vs Background workloads:

Carryforward Debt

When a workload consumes more CUs than the capacity has available, a "carryforward debt" accumulates. Fabric uses this debt to throttle future operations proportionally. High carryforward debt means subsequent workloads will be throttled even if no active workloads are running.

Capacity: F64 (64 CUs)
Heavy Spark job runs: consumes 500 CU-seconds in 10 seconds
Debt accumulated: 500 - 640 (10s × 64 CUs) = -140 CU-seconds carryforward debt
Recovery time: ~2 minutes at idle before full capacity is available again

4. Capacity REST API

# Get capacity details
curl "https://api.fabric.microsoft.com/v1/capacities/${CAPACITY_ID}" \
  -H "Authorization: Bearer ${TOKEN}"

# Pause a capacity (billing stops)
curl -X POST "https://api.fabric.microsoft.com/v1/capacities/${CAPACITY_ID}/suspend" \
  -H "Authorization: Bearer ${TOKEN}"

# Resume a capacity
curl -X POST "https://api.fabric.microsoft.com/v1/capacities/${CAPACITY_ID}/resume" \
  -H "Authorization: Bearer ${TOKEN}"

# Resize capacity (e.g., F32 → F64 for peak hours)
curl -X PATCH "https://api.fabric.microsoft.com/v1/capacities/${CAPACITY_ID}" \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"sku": {"name": "F64", "tier": "Fabric"}}'

# Assign workspace to capacity
curl -X POST "https://api.fabric.microsoft.com/v1/workspaces/${WORKSPACE_ID}/assignToCapacity" \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"capacityId": "'${CAPACITY_ID}'"}'

Capacity Admin API (requires Power BI admin token):

# List all capacities in tenant
curl "https://api.powerbi.com/v1.0/myorg/admin/capacities" \
  -H "Authorization: Bearer ${ADMIN_TOKEN}"

# Get workloads on a capacity
curl "https://api.powerbi.com/v1.0/myorg/admin/capacities/${CAPACITY_ID}/workloads" \
  -H "Authorization: Bearer ${ADMIN_TOKEN}"

# Update workload settings (e.g., increase memory for Spark)
curl -X PATCH "https://api.powerbi.com/v1.0/myorg/admin/capacities/${CAPACITY_ID}/workloads/dataflow" \
  -H "Authorization: Bearer ${ADMIN_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"state": "Enabled", "maxMemoryPercentageSetByUser": 40}'

5. Capacity Metrics App

The Fabric Capacity Metrics app is the primary tool for monitoring CU consumption. Install it from AppSource: search "Microsoft Fabric Capacity Metrics" in the Microsoft AppSource.

Key Metrics

KQL Queries for Metrics App

// Top 10 operations by CU consumption (last 24 hours)
capacityOperations
| where TimeGenerated > ago(24h)
| summarize TotalCU = sum(TotalCUSeconds) by OperationName, WorkspaceName, ItemKind
| top 10 by TotalCU desc
| project WorkspaceName, ItemKind, OperationName, TotalCU = round(TotalCU, 2)

// CU consumption by workspace
capacityOperations
| where TimeGenerated > ago(7d)
| summarize TotalCU = sum(TotalCUSeconds) by WorkspaceName
| order by TotalCU desc

// Throttling events over time
capacityMetrics
| where TimeGenerated > ago(24h)
| where MetricName == "ThrottlingPercentage"
| summarize MaxThrottling = max(MetricValue), AvgThrottling = avg(MetricValue)
    by bin(TimeGenerated, 15m)
| where MaxThrottling > 0
| render timechart

// Direct Lake fallback frequency (important for semantic model performance)
capacityOperations
| where TimeGenerated > ago(24h)
| where OperationName == "DirectLakeFallback"
| summarize FallbackCount = count() by WorkspaceName, ItemName
| order by FallbackCount desc

6. Throttling Diagnosis

Throttling Symptoms

Identify Peak Usage Windows

// Heatmap: CU usage by hour of day and day of week
capacityOperations
| where TimeGenerated > ago(30d)
| extend HourOfDay = hourofday(TimeGenerated), DayOfWeek = dayofweek(TimeGenerated)
| summarize AvgCU = avg(TotalCUSeconds) by HourOfDay, DayOfWeek
| order by DayOfWeek asc, HourOfDay asc

// Find schedule collisions — jobs starting at the same time
capacityOperations
| where TimeGenerated > ago(7d)
| where OperationName in ("DataPipelineRun", "SemanticModelRefresh", "SparkJobRun")
| summarize JobCount = count() by bin(TimeGenerated, 5m)
| where JobCount > 5  // More than 5 concurrent scheduled jobs
| order by TimeGenerated desc

Throttling Investigation Checklist

1. Check Fabric Capacity Metrics app for throttling events (last 24h)
2. Identify which workspaces / operations caused the CU spike
3. Check for schedule collisions (multiple heavy jobs starting at the same time)
4. Check carryforward debt recovery time
5. Assess: is this a recurring pattern or a one-time event?
6. Options:
   a. Stagger schedules to distribute CU load
   b. Reduce parallelism in Spark jobs (fewer executors)
   c. Move non-critical workloads to off-peak hours
   d. Upgrade SKU (increase CUs)
   e. Split workloads across multiple capacities

7. Workload Tuning

Spark Notebook Optimization

# Reduce executor count for low-priority background jobs
spark.conf.set("spark.executor.instances", "2")  # Down from default 4-8
spark.conf.set("spark.executor.cores", "2")

# Use Delta table optimization features to reduce Spark CU consumption
# 1. V-Order on write (Fabric default — keep enabled for best read performance)
spark.conf.set("spark.microsoft.delta.optimizeWrite.enabled", "true")
spark.conf.set("spark.microsoft.delta.optimizeWrite.binSize", "1073741824")  # 1 GB

# 2. Auto-compact small files (reduces future read CU)
spark.conf.set("spark.databricks.delta.autoCompact.enabled", "true")
spark.conf.set("spark.databricks.delta.autoCompact.minNumFiles", "50")

# 3. Use ZORDER for frequently filtered columns (reduces scan CU)
spark.sql("OPTIMIZE sales_lakehouse.FactSales ZORDER BY (CustomerId, OrderDate)")

Schedule Staggering

# Python script to stagger pipeline refresh schedules
# Distribute 12 semantic model refreshes across 3-hour window
import datetime

BASE_HOUR = 5  # Start at 5 AM
INTERVAL_MINUTES = 15  # 15 minutes between each refresh

refresh_times = []
for i in range(12):
    offset = i * INTERVAL_MINUTES
    hour = BASE_HOUR + offset // 60
    minute = offset % 60
    refresh_times.append(f"{hour:02d}:{minute:02d}")

print("Recommended refresh schedule:")
for i, t in enumerate(refresh_times):
    print(f"  Model {i+1}: {t}")

Pipeline Parallelism Control

// In Data Pipeline — reduce concurrent activity runs
{
  "name": "ForEachTable",
  "type": "ForEach",
  "properties": {
    "isSequential": false,
    "batchCount": 3,  // Run only 3 tables in parallel (default is 20)
    "activities": [...]
  }
}

Semantic Model Refresh Optimization

# Enhanced refresh with controlled parallelism
curl -X POST \
  "https://api.powerbi.com/v1.0/myorg/groups/${WORKSPACE_ID}/datasets/${DATASET_ID}/refreshes" \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "full",
    "commitMode": "transactional",
    "maxParallelism": 3,    // Limit parallel table processing (default: up to 16)
    "retryCount": 1,
    "notifyOption": "MailOnFailure"
  }'

8. Autoscale Planning

Autoscale for Fabric Capacities

Fabric does not natively support automatic SKU scaling (unlike Azure Analysis Services). However, you can implement scheduled scaling using the Fabric REST API and Azure automation:

# Azure Function / Logic App — Scale up for business hours, scale down for nights
param($timer)

$TenantId = $env:FABRIC_TENANT_ID
$ClientId = $env:FABRIC_CLIENT_ID
$ClientSecret = $env:FABRIC_CLIENT_SECRET
$CapacityId = $env:CAPACITY_ID

# Get access token
$tokenBody = @{
    grant_type    = "client_credentials"
    client_id     = $ClientId
    client_secret = $ClientSecret
    scope         = "https://api.fabric.microsoft.com/.default"
}
$token = (Invoke-RestMethod -Uri "https://login.microsoftonline.com/$TenantId/oauth2/v2.0/token" -Method POST -Body $tokenBody).access_token
$headers = @{ "Authorization" = "Bearer $token"; "Content-Type" = "application/json" }

# Determine target SKU based on current hour (UTC)
$hour = (Get-Date).ToUniversalTime().Hour
$targetSku = if ($hour -ge 7 -and $hour -lt 20) { "F64" } else { "F16" }

# Scale capacity
$body = @{ sku = @{ name = $targetSku; tier = "Fabric" } } | ConvertTo-Json
Invoke-RestMethod -Uri "https://api.fabric.microsoft.com/v1/capacities/$CapacityId" `
  -Method PATCH -Headers $headers -Body $body

Write-Host "Scaled capacity to $targetSku at $((Get-Date).ToUniversalTime())"

Cost Estimation by SKU

Costs are approximate and vary by Azure region. Check Azure pricing calculator for current rates.

Pause/Resume savings:

• Pausing a capacity stops billing.
• Resume time: 2–5 minutes for smaller SKUs; up to 10 minutes for F512+.
• For non-business-hours savings: pause F64 for 12 hours/day = ~50% cost reduction.

9. Common Workflows

Workflow 1: Capacity Upgrade for Peak Period

# 1. Check current utilization
curl "https://api.fabric.microsoft.com/v1/capacities/${CAPACITY_ID}" \
  -H "Authorization: Bearer ${TOKEN}"

# 2. Scale up before peak
curl -X PATCH "https://api.fabric.microsoft.com/v1/capacities/${CAPACITY_ID}" \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"sku": {"name": "F128", "tier": "Fabric"}}'

# 3. Monitor in Capacity Metrics app during peak
# 4. Scale back down after peak period
curl -X PATCH "https://api.fabric.microsoft.com/v1/capacities/${CAPACITY_ID}" \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"sku": {"name": "F64", "tier": "Fabric"}}'

Workflow 2: Investigate and Resolve Throttling

1. Open Fabric Capacity Metrics app
2. Filter to last 24h — identify the throttling window
3. Drill into the throttling period:
   - Which workspaces were active?
   - Which operation types (Spark, Pipeline, Refresh)?
4. Check concurrent operations — were multiple heavy jobs running simultaneously?
5. Apply remediations:
   a. Stagger: spread schedules by 15-30 minutes
   b. Limit: reduce maxParallelism on Enhanced Refresh
   c. Defer: move low-priority jobs to overnight
   d. Upgrade: if consistently > 80% utilization, upgrade SKU
6. Monitor for 1 week to confirm improvement

10. Error Handling and Troubleshooting

11. Performance and Limits

Progressive Disclosure — Reference Files