Azure Cost Analysis
Diagnostic toolkit for FXCI Azure CI cost investigations. Covers all 3 CI subscriptions and multiple cost dimensions (pool, SKU, region, service). Designed for both routine cost trend analysis and deep diagnostic investigations.
Knowledge References
Read first:
@references/README.md
@references/user-inputs.md
@references/methodology.md
Multi-dimensional and multi-subscription:
@references/cost-dimensions.md
@references/multi-subscription.md
API and task correlation:
@references/cost-management-api.md
@references/taskcluster-task-counting.md
For deep diagnostics:
@references/fxci-config-lookup.md
@references/taskcluster-service-changes.md
@references/azure-spot-pricing.md
@references/spot-evictions.md
Standalone SQL queries (copy-paste-ready, one file per query):
queries/azure-task-counts.sqlqueries/spot-evictions.sqlqueries/retry-rate.sqlqueries/schema-sample.sql
Prerequisites
- Azure CLI (
az) authenticated, with read access to the 3 CI subscriptions
- Python 3.10+
curl (for Taskcluster API queries)- Redash access (for
taskclusteretl.derived_task_summary queries when correlating cost with tasks)
Quick Start
Routine: monthly cost trend
# Monthly costs by worker-pool-id, FXCI Azure DevTest
uv run scripts/query_costs.py --start 2026-01-01 --end 2026-03-31 --granularity monthly
# Compare two periods side by side
uv run scripts/query_costs.py --start 2026-01-01 --end 2026-03-31 --granularity monthly --compare-months
# Different subscription
uv run scripts/query_costs.py --start 2026-03-01 --end 2026-03-31 --subscription a30e97ab-734a-4f3b-a0e4-c51c0bff0701
Diagnostic: investigating a cost increase
- Read methodology.md first — picks daily-cost vs cost/task as your primary signal
- Run
query_costs.py per subscription for the period of interest, monthly + daily
- Check TC Engineering DevTest as a control — if it didn't grow, rule out general Azure pricing event
- Group by Meter in the Azure Portal Cost Analysis to see SKU-level breakdown
- Group by Resource location if you suspect regional spot pricing changes — pair with
azure-spot-pricing.md for the Retail Prices API
- Pull
derived_task_summary task data from Redash for the same window — see taskcluster-task-counting.md BigQuery section
- Verify pool config via
fxci-config-lookup.md — SKU, regions, maxCapacity, initialWeight haven't changed
- Check git history of
worker-pools.yml for cost-relevant changes during the window
- Check TC services repo for cost-relevant changes —
taskcluster-service-changes.md. Bugs in worker-manager/worker-scanner can drive cost in ways fxci-config can't explain.
- Check the TC issues tracker for any bug reports filed around the cost-anomaly window — engineering may have observed symptoms before you did
- If pattern is broad cost/task increase across regions/pools — investigate spot eviction (
spot-evictions.md)
- If queue starvation observed alongside available capacity — likely a TC service-level state issue; cross-check
taskcluster-service-changes.md
Comparing task counts for a specific push (chunk count investigations)
uv run scripts/count_push_tasks.py --date 2026.03.30
Usage
query_costs.py
Queries the Azure Cost Management REST API for actual costs grouped by worker-pool-id.
| Flag |
Description |
--start |
Start date (YYYY-MM-DD), required |
--end |
End date (YYYY-MM-DD), required |
--granularity |
monthly or daily (default: monthly) |
--compare-months |
Show month-over-month deltas and identify top movers |
--top |
Number of top pools to display (default: 25) |
--output, -o |
Save raw API response as JSON |
--subscription |
Override subscription ID (default: FXCI Azure DevTest). Use Trusted FXCI: a30e97ab-734a-4f3b-a0e4-c51c0bff0701, TC Engineering: 8a205152-b25a-417f-a676-80465535a6c9 |
count_push_tasks.py
Counts tasks per worker pool and test suite in a mozilla-central push task group. Use this for drilling into a single push (e.g. chunk count investigations). For batch analysis across many pushes/days, use the BigQuery approach in taskcluster-task-counting.md.
| Flag |
Description |
--date |
Push date in TC index format: YYYY.MM.DD |
--push-index |
Which push on that date to analyze (default: 0 = first) |
--pool-filter |
Only show tasks matching this pool substring |
Example Prompts
| Prompt |
Action |
| "Why did Azure costs go up this month?" |
Read methodology.md. Run query_costs.py --compare-months per subscription. Check TC Engineering as control. Group by Meter to identify SKU drivers. |
| "Show me daily costs for March" |
query_costs.py --start 2026-03-01 --end 2026-03-31 --granularity daily |
| "Did per-task billing rate change?" |
Compute volume-weighted cost/task by week using F8s v2 cost ÷ F8s v2 pool tasks. Read methodology.md for the cost/task amortization caveat. |
| "Which worker pools are most expensive?" |
query_costs.py for the period, sorted by total spend. |
| "Are there new worker pools driving cost?" |
query_costs.py --compare-months, look at the "new pools" section. |
| "Did test task volume increase?" |
Pull taskclusteretl.derived_task_summary task counts via Redash for the period. See taskcluster-task-counting.md. |
| "Did spot prices change in our regions?" |
Query Azure Retail Prices API for current SKU prices and effective dates. See azure-spot-pricing.md. |
| "Are spot evictions causing the cost rise?" |
Pull eviction events from Azure Activity Log + worker-manager preemption events. See spot-evictions.md. |
| "Which test suites are running more on win11-64-24h2?" |
count_push_tasks.py --pool-filter win11-64-24h2 for two dates, compare suite counts. |
| "Did the VM SKU for this pool change?" |
Check worker-pools.yml directly + git log via GitHub API. See fxci-config-lookup.md. |
| "Is the cost increase Azure-wide or CI-specific?" |
Compare cost growth in TC Engineering DevTest vs CI subs. If TC Engineering is flat, it's CI-specific. |
Workflow: Monthly Cost Review
- Query monthly costs for the period of interest with
--compare-months, all 3 subscriptions
- Identify top movers — pools with the largest absolute increase
- Check for new pools — pools that didn't exist in the prior period
- Compute cost/task for stable-volume pools to detect rate vs volume changes
- Correlate with task volume — use
count_push_tasks.py for spot checks, BigQuery for batch
- Drill into test suites — if a pool's cost grew, check which test suites gained the most tasks
- Save report to
~/moz_artifacts/ with findings
Workflow: Diagnostic Investigation
For deeper cost investigations (cost rising more than volume can explain):
- Establish a clean baseline (a normal pre-anomaly month — see
methodology.md)
- Compute volume-weighted daily cost/task; find the inflection date
- Rule out, with evidence: general Azure pricing event (TC Engineering control), single-region spot spike, OS/SKU migration, regional traffic shifts, volume alone
- Check fxci-config git history for changes in the inflection window
- Check taskcluster/taskcluster git history for service-level changes (worker-manager, worker-scanner) in the window
- Check the taskcluster/taskcluster issues tracker for bug reports filed around the window
- Pull spot price effective dates from Retail Prices API
- If pattern is broad cost/task increase across pools/regions: pull spot eviction telemetry from
fxci.task_runs
- If queue starvation observed alongside available capacity: investigate TC service state-tracking bugs
- If structural changes (maxCapacity, initialWeight) candidate: design a rollback experiment
1---2name: azure-cost-analysis3description: Analyze FXCI Azure CI costs across the 3 CI subscriptions (FXCI DevTest, Trusted FXCI, TC Engineering). Investigate cost increases by worker pool, SKU, region, or service. Distinguish volume-driven changes from rate changes (spot pricing, eviction overhead, VM lifecycle). Correlate with Taskcluster task volume from BigQuery and check fxci-config for cost- relevant changes. Triggers on "azure cost", "cost increase", "cost analysis", "worker pool cost", "FXCI spend", "cloud spend", "cost trend", "why did cost go up", "cost by worker pool", "cost per task", "spot eviction", "spot pricing", "cost diagnostic".4---56# Azure Cost Analysis78Diagnostic toolkit for FXCI Azure CI cost investigations. Covers all 3 CI subscriptions and multiple cost dimensions (pool, SKU, region, service). Designed for both routine cost trend analysis and deep diagnostic investigations.910## Knowledge References1112**Read first:**13@references/README.md14@references/user-inputs.md15@references/methodology.md1617**Multi-dimensional and multi-subscription:**18@references/cost-dimensions.md19@references/multi-subscription.md2021**API and task correlation:**22@references/cost-management-api.md23@references/taskcluster-task-counting.md2425**For deep diagnostics:**26@references/fxci-config-lookup.md27@references/taskcluster-service-changes.md28@references/azure-spot-pricing.md29@references/spot-evictions.md3031**Standalone SQL queries** (copy-paste-ready, one file per query):32- `queries/azure-task-counts.sql`33- `queries/spot-evictions.sql`34- `queries/retry-rate.sql`35- `queries/schema-sample.sql`3637## Prerequisites3839- Azure CLI (`az`) authenticated, with read access to the 3 CI subscriptions40- Python 3.10+41- `curl` (for Taskcluster API queries)42- Redash access (for `taskclusteretl.derived_task_summary` queries when correlating cost with tasks)4344## Quick Start4546### Routine: monthly cost trend47```bash48# Monthly costs by worker-pool-id, FXCI Azure DevTest49uv run scripts/query_costs.py --start 2026-01-01 --end 2026-03-31 --granularity monthly5051# Compare two periods side by side52uv run scripts/query_costs.py --start 2026-01-01 --end 2026-03-31 --granularity monthly --compare-months5354# Different subscription55uv run scripts/query_costs.py --start 2026-03-01 --end 2026-03-31 --subscription a30e97ab-734a-4f3b-a0e4-c51c0bff070156```5758### Diagnostic: investigating a cost increase59601. **Read methodology.md first** — picks daily-cost vs cost/task as your primary signal612. **Run `query_costs.py` per subscription** for the period of interest, monthly + daily623. **Check TC Engineering DevTest** as a control — if it didn't grow, rule out general Azure pricing event634. **Group by Meter** in the Azure Portal Cost Analysis to see SKU-level breakdown645. **Group by Resource location** if you suspect regional spot pricing changes — pair with `azure-spot-pricing.md` for the Retail Prices API656. **Pull `derived_task_summary`** task data from Redash for the same window — see `taskcluster-task-counting.md` BigQuery section667. **Verify pool config** via `fxci-config-lookup.md` — SKU, regions, maxCapacity, initialWeight haven't changed678. **Check git history** of `worker-pools.yml` for cost-relevant changes during the window689. **Check TC services repo** for cost-relevant changes — `taskcluster-service-changes.md`. Bugs in worker-manager/worker-scanner can drive cost in ways fxci-config can't explain.6910. **Check the TC issues tracker** for any bug reports filed around the cost-anomaly window — engineering may have observed symptoms before you did7011. **If pattern is broad cost/task increase across regions/pools** — investigate spot eviction (`spot-evictions.md`)7112. **If queue starvation observed alongside available capacity** — likely a TC service-level state issue; cross-check `taskcluster-service-changes.md`7273### Comparing task counts for a specific push (chunk count investigations)74```bash75uv run scripts/count_push_tasks.py --date 2026.03.3076```7778## Usage7980### query_costs.py8182Queries the Azure Cost Management REST API for actual costs grouped by `worker-pool-id`.8384| Flag | Description |85|------|-------------|86| `--start` | Start date (YYYY-MM-DD), required |87| `--end` | End date (YYYY-MM-DD), required |88| `--granularity` | `monthly` or `daily` (default: `monthly`) |89| `--compare-months` | Show month-over-month deltas and identify top movers |90| `--top` | Number of top pools to display (default: 25) |91| `--output`, `-o` | Save raw API response as JSON |92| `--subscription` | Override subscription ID (default: FXCI Azure DevTest). Use Trusted FXCI: `a30e97ab-734a-4f3b-a0e4-c51c0bff0701`, TC Engineering: `8a205152-b25a-417f-a676-80465535a6c9` |9394### count_push_tasks.py9596Counts tasks per worker pool and test suite in a mozilla-central push task group. Use this for drilling into a single push (e.g. chunk count investigations). For batch analysis across many pushes/days, use the BigQuery approach in `taskcluster-task-counting.md`.9798| Flag | Description |99|------|-------------|100| `--date` | Push date in TC index format: `YYYY.MM.DD` |101| `--push-index` | Which push on that date to analyze (default: 0 = first) |102| `--pool-filter` | Only show tasks matching this pool substring |103104## Example Prompts105106| Prompt | Action |107|--------|--------|108| "Why did Azure costs go up this month?" | Read `methodology.md`. Run `query_costs.py --compare-months` per subscription. Check TC Engineering as control. Group by Meter to identify SKU drivers. |109| "Show me daily costs for March" | `query_costs.py --start 2026-03-01 --end 2026-03-31 --granularity daily` |110| "Did per-task billing rate change?" | Compute volume-weighted cost/task by week using F8s v2 cost ÷ F8s v2 pool tasks. Read `methodology.md` for the cost/task amortization caveat. |111| "Which worker pools are most expensive?" | `query_costs.py` for the period, sorted by total spend. |112| "Are there new worker pools driving cost?" | `query_costs.py --compare-months`, look at the "new pools" section. |113| "Did test task volume increase?" | Pull `taskclusteretl.derived_task_summary` task counts via Redash for the period. See `taskcluster-task-counting.md`. |114| "Did spot prices change in our regions?" | Query Azure Retail Prices API for current SKU prices and effective dates. See `azure-spot-pricing.md`. |115| "Are spot evictions causing the cost rise?" | Pull eviction events from Azure Activity Log + worker-manager preemption events. See `spot-evictions.md`. |116| "Which test suites are running more on win11-64-24h2?" | `count_push_tasks.py --pool-filter win11-64-24h2` for two dates, compare suite counts. |117| "Did the VM SKU for this pool change?" | Check `worker-pools.yml` directly + git log via GitHub API. See `fxci-config-lookup.md`. |118| "Is the cost increase Azure-wide or CI-specific?" | Compare cost growth in TC Engineering DevTest vs CI subs. If TC Engineering is flat, it's CI-specific. |119120## Workflow: Monthly Cost Review1211221. **Query monthly costs** for the period of interest with `--compare-months`, all 3 subscriptions1232. **Identify top movers** — pools with the largest absolute increase1243. **Check for new pools** — pools that didn't exist in the prior period1254. **Compute cost/task** for stable-volume pools to detect rate vs volume changes1265. **Correlate with task volume** — use `count_push_tasks.py` for spot checks, BigQuery for batch1276. **Drill into test suites** — if a pool's cost grew, check which test suites gained the most tasks1287. **Save report** to `~/moz_artifacts/` with findings129130## Workflow: Diagnostic Investigation131132For deeper cost investigations (cost rising more than volume can explain):1331341. Establish a clean baseline (a normal pre-anomaly month — see `methodology.md`)1352. Compute volume-weighted daily cost/task; find the inflection date1363. Rule out, with evidence: general Azure pricing event (TC Engineering control), single-region spot spike, OS/SKU migration, regional traffic shifts, volume alone1374. Check **fxci-config** git history for changes in the inflection window1385. Check **taskcluster/taskcluster** git history for service-level changes (worker-manager, worker-scanner) in the window1396. Check the **taskcluster/taskcluster issues tracker** for bug reports filed around the window1407. Pull spot price effective dates from Retail Prices API1418. If pattern is broad cost/task increase across pools/regions: pull spot eviction telemetry from `fxci.task_runs`1429. If queue starvation observed alongside available capacity: investigate TC service state-tracking bugs14310. If structural changes (maxCapacity, initialWeight) candidate: design a rollback experiment
mozilla-platform-ops