# Oncall Weekly Telemetry Report

> Generate the weekly Android on-call (OCE) WoW + 60-day trend telemetry reports as polished self-contained HTML. Produces TWO reports by default — the Android Broker report (from `android_spans` materialized views) and the Authenticator app report (from the Authenticator scenario materialized views) — plus a combined index page. Use this skill for the weekly OCE rotation when asked to "produce the OCE report", "weekly on-call report", "WoW telemetry report", "weekly broker health report", "authenticator weekly report", "authapp telemetry report", or "generate this week's on-call summary". Supports modes `both` (default), `broker`, and `authapp`. Writes to `$env:USERPROFILE\android-oce-reports\` (outside the workspace so reports are never committed).

- Skill: `azuread/oncall-weekly-telemetry-report` (Agent Skill, multi-file: 49 files)
- Install (CLI): `npx skillmds@latest add azuread/oncall-weekly-telemetry-report`
- Raw SKILL.md: https://api.skillmd.com/api/skills/azuread/oncall-weekly-telemetry-report/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Web & Frontend
- Author: AzureAD (https://skillmd.com/u/azuread)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/azuread/oncall-weekly-telemetry-report

---


# OCE Weekly Report — router

This skill produces the weekly Android on-call telemetry reports. It is a **router**: it resolves
the reporting window, decides which app playbook(s) to run, and stitches the results together into
a combined index. **All app-specific analysis lives in the playbooks** — read the one(s) you need
after resolving the mode.

| Playbook | Covers | Read when mode is |
|---|---|---|
| [`assets/playbooks/broker.md`](assets/playbooks/broker.md) | Android Broker — error codes/types, spike + code attribution, latency, broker version adoption | `broker`, `both` |
| [`assets/playbooks/authapp.md`](assets/playbooks/authapp.md) | Authenticator app — scenario funnels (Passkey / Entra MFA / Entra PSI / MSA NGC+SA), error reasons, abandonment, Broker API responsiveness, crash rate | `authapp`, `both` |

> **Do not read both playbooks into one context when running `both`.** Run them as two parallel
> sub-agents (see § Orchestration). Each playbook is large by design; interleaving them degrades
> both reports and risks cross-contaminating the two apps' incompatible Kusto conventions.

---

## Mode selection

**Default is `both`.** Infer the mode from the request; only ask if genuinely ambiguous.

| Signal in the request | Mode |
|---|---|
| *"the OCE report"*, *"weekly on-call report"*, *"this week's telemetry"*, no app named | `both` |
| *"broker report"*, *"broker health"*, *"error codes"*, *"spike attribution"* | `broker` |
| *"authenticator report"*, *"authapp"*, *"passkey/MFA/PSI/MSA scenarios"*, *"registration success rate"* | `authapp` |

Optional flags the user may add: `--skip-crashes` (Authenticator only — skips the App Center
crash layer, which needs a secret), `--end YYYY-MM-DD` (see § Reporting window).

---

## Reporting window (shared by both apps)

> **⚠️ Do NOT ask the user for a reporting date by default.** Both reports use the **same rolling
> 7-day window** ending at start-of-day UTC on the invocation day, so the most recent complete days
> are always captured and the two reports are directly comparable.
> [`assets/scripts/bootstrap-report.ps1`](assets/scripts/bootstrap-report.ps1) computes and stamps
> the window automatically. The previous "confirm the Sunday bucket with the user" flow silently
> produced stale windows (a Thursday run emitted a 4-day partial window and dropped the last
> complete week) — that failure mode is what this section explicitly guards against.

1. **Reporting window** — do NOT ask. Run `bootstrap-report.ps1`; it resolves the window silently
   and prints:
   ```
   Resolved reporting window (UTC):   # example values for a run on 2026-07-15
     Last 7 days:   2026-07-08 -> 2026-07-15  (exclusive upper bound)
     Baseline:      2026-07-01 -> 2026-07-08
     60-day trend:  2026-05-16 -> 2026-07-15  (literal 60d ending today; chart includes current partial week)
     Trend delta cutoff: weeks < 2026-07-12  (startofweek(curEnd); pass as bucket-trends.js --end)
   ```
   These dates are stamped into each report's `<title>`, `<div class="meta">`, and Generated banner
   during bootstrap — you do not hand-edit them.

   **In `both` mode, resolve the window ONCE and pass the same `-EndDate` to both bootstrap calls.**
   Letting each playbook resolve its own window independently is how the two reports drift onto
   different days when a run straddles UTC midnight.

   **Override only when the user explicitly requests a non-default window.** Signals: "the week of
   X", "as of last Friday", "the report from three weeks ago", "in-progress data", "just today's
   numbers". Then use:
   ```pwsh
   .\bootstrap-report.ps1 -App broker -EndDate 2026-07-02   # e.g. reproduce the report as of Jul 2
   ```
   `-EndDate` is the exclusive upper bound of the current window (`curEnd`); the script derives
   `curStart = curEnd - 7d` and `prevStart = curEnd - 14d` deterministically. `-EndDate` must be
   today or earlier — future dates are refused.

   **Supported override = `-EndDate` only (shifts the window *end*).** The 7-day primary span, the
   7-day baseline, and the 60-day trend form a fixed frame that moves *rigidly* with `-EndDate`.
   There is **no** custom start-date and **no** arbitrary-span flag — a request like "last 30 days"
   or "from Jun 1 to Jun 20" is **not** expressible via a single parameter. To produce a longer or
   custom span you must edit the KQL window placeholders (`<CUR_START>` / `<CUR_END>` /
   `<PREV_START>`; the baseline end is always `<CUR_START>`, so there is no separate `<PREV_END>`
   token) by hand. If the user asks for a non-7-day span, say so up front rather than silently
   emitting a 7-day report.

2. **Comparison baseline** — auto-computed as the immediately-prior 7 days
   (`[curEnd - 14d, curEnd - 7d)`). No user input.

3. **60-day trend window** — auto-computed as the **literal last 60 days ending today**
   (`[curEnd - 60d, curEnd)`), so both bounds move with `-EndDate`. Trend sections are Sun-Sat
   weekly-bucketed (Kusto `startofweek()` is Sunday-aligned) because the trend needs stable weekly
   denominators, but the final bar is the **current in-progress (partial) week** — the chart ends
   today. Regression/improvement **delta classification is still computed on complete weeks only**
   (`bucket-trends.js --end=startofweek(curEnd) --include-partial-end`); a partial week as "last"
   would read as a fake −99% improvement, so it is charted but excluded from the delta math.

**Kusto note (weekly-bucketed queries only):** `startofweek()` is Sunday-aligned, so
`startofweek('2026-05-09') == 2026-05-03T00:00:00Z`. When authoring weekly-bucketed queries, verify
by printing the distinct week values from your first query. Off-by-one-week is the #1 silent error
in weekly-bucket queries.

---

## Outputs

All outputs land in `$env:USERPROFILE\android-oce-reports\` — **outside the workspace**, so reports
can never be committed accidentally.

| Mode | Files produced |
|---|---|
| `broker` | `oncall-wow-report-<curEnd>.html` |
| `authapp` | `authapp-wow-report-<curEnd>.html` |
| `both` | both of the above **plus** `oce-index-<curEnd>.html` |

`<curEnd>` is `YYYY-MM-DD`, the end-date of the rolling 7-day window. Raw KQL payloads are cached
under `_data/<app>-<curEnd>/` so each report is reproducible; folders older than 60 days are pruned
automatically by `bootstrap-report.ps1`.

### The combined index (`both` mode only)

After both reports validate, build the index:

```pwsh
.\.github\skills\oncall-weekly-telemetry-report\assets\scripts\build-index.ps1 -EndDate <curEnd>
```

It reads the headline KPI tiles out of both finished reports and emits a single one-page digest
with links to each. It is a **digest, not an analysis** — do not write new findings into it that
aren't already in one of the two reports. If a cross-app finding matters (e.g. an Authenticator
Broker-API responsiveness spike that maps to a Broker error code), write it in **both** reports and
let the index link them.

---

## Orchestration (`both` mode)

Run the two playbooks **in parallel as background sub-agents**, not sequentially. Serially, `both`
roughly doubles an already-long run; in parallel it costs about as much wall-clock as one report.

1. Resolve the window once (`bootstrap-report.ps1 -App broker` prints it; capture `curEnd`).
2. Bootstrap **both** report files up front with the same `-EndDate`, so neither agent can drift:
   ```pwsh
   $S = '.\.github\skills\oncall-weekly-telemetry-report\assets\scripts'
   & "$S\bootstrap-report.ps1" -App broker  -EndDate <curEnd>
   & "$S\bootstrap-report.ps1" -App authapp -EndDate <curEnd>
   ```
3. Launch two `general-purpose` background agents. Give each the **full** context it needs — the
   playbook path, the resolved window dates, its bootstrapped report path, and its `_data` folder.
   Require each agent to read **both** its playbook **and** § Shared hard rules in this file before
   writing any HTML. Instruct each to do the work itself (not to advise), and to run its own
   validator before reporting back.
4. Wait for both. Do not start the index until both validators pass.
5. Run `build-index.ps1`.
6. Report the three file paths to the user in chat. Do **not** paste report contents into chat.

If one agent fails, still publish the other report and say plainly in chat which app failed and
why — a half-delivered rotation report beats a blocked one.

---

## Shared assets

| File | Purpose |
|---|---|
| [`scripts/bootstrap-report.ps1`](assets/scripts/bootstrap-report.ps1) | Bootstrap a report from its app's canonical template. `-App broker\|authapp`. Resolves + stamps the rolling window, creates `_data/<app>-<curEnd>/`, prunes old data, detects stub-vs-real collisions. |
| [`scripts/run-kql.ps1`](assets/scripts/run-kql.ps1) | Direct-REST Kusto helper — drop-in fallback when the Kusto MCP times out. `-App broker\|authapp` selects the cluster + database; `-Cluster`/`-Database` still override explicitly. |
| [`scripts/validate-report.ps1`](assets/scripts/validate-report.ps1) | Pre-publish validator. `-App broker\|authapp` selects the check profile. Shared checks (stale tokens, mojibake, div balance, sparkline coverage, header/filename date agreement) run for both. |
| [`scripts/build-index.ps1`](assets/scripts/build-index.ps1) | Build the combined `oce-index-<curEnd>.html` digest from two finished reports. |
| [`scripts/bucket-trends.js`](assets/scripts/bucket-trends.js) | Bucket any `{key, week, metric}` series into 60-day regression / spike / improvement / flat. App-agnostic — `--key=` selects the grouping column. |
| [`scripts/classify-novelty.js`](assets/scripts/classify-novelty.js) | Reads a `bucket-trends.js --json=` sidecar and labels each key **NEW / ACCELERATING / ONGOING / VOLATILE / RECOVERY / IMPROVING / STABLE** against its own 7-week baseline, plus family clustering. App-agnostic. **Mandatory in both playbooks** — it is what stops the attention section from being a volume-ranked list where a flat-but-huge code outranks a real step change, and it is also the **noise gate**: its `attention` set (`NEW` + `ACCELERATING`), plus at most 2 wins, is all that renders visibly with charts — everything else collapses into a fold. |
| [`scripts/agg.js`](assets/scripts/agg.js) | Per-key per-dim top-N rollup with WoW deltas. |
| [`scripts/find-suspect-prs.ps1`](assets/scripts/find-suspect-prs.ps1) | Parallel `git log -S` + `--grep` for a symbol. `-Repos` selects which repos to scan (broker/common, or authenticator). |
| [`scripts/visual-smoke.ps1`](assets/scripts/visual-smoke.ps1) | Optional Playwright layout smoke test — catches rendered-layout bugs (text bleed, cards touching) that HTML/CSS validation can't see. |
| [`templates/index-template.html`](assets/templates/index-template.html) | Canonical layout for the combined index page. |

App-specific assets (queries, cheatsheets, report templates) are listed in each playbook.

---

## Shared hard rules

These apply to **both** reports. App-specific hard rules (HLL device counting, TDigest percentiles,
`Merge*` helpers, the Originator pre-check, Authenticator `*DCount` columns, volume floors) live in
the playbooks — and **they are not interchangeable**. The Broker's "never `sum(countDevices)`" rule
is *actively wrong* on the Authenticator side, where `sum(SucceededDCount)` is the correct idiom.
Never carry a convention across the two playbooks.

- **Never carry a numeric telemetry value forward between runs.** Every KPI, table cell, delta %,
  device/request count, sparkline point, and verdict number must be re-pulled from Kusto for *this*
  run — never copied from a previous report, from a checkpoint/summary, from notes, or from memory.
  Telemetry shifts between runs and stale numbers read as fabricated. Near-miss precedent: a
  `no_tokens_found` count was about to be carried as ~23.7M when the actual current-window value
  was ~4.86M — a ~5× error that only the re-pull caught. If a number isn't backed by a query result
  file in this run's `_data/<app>-<curEnd>/`, it does not go in the report.
- **Never hardcode the "Generated" date.** It is the *run* date in **UTC**, auto-stamped by
  `bootstrap-report.ps1`. If you rebuild a body programmatically, derive it live with a **UTC-date**
  formatter (`new Date().toISOString().slice(0,10)` in Node,
  `[datetime]::UtcNow.ToString('yyyy-MM-dd')` in PowerShell) — never paste a literal.
- **⚠️ UTF-8 trap — DO NOT use PowerShell `@'...'@` heredocs to compose HTML containing emojis,
  em-dashes, arrows, or middle dots.** PowerShell silently strips multi-byte UTF-8 characters when
  piping heredocs to `Set-Content` / `Out-File`. Use Node (`fs.writeFileSync`),
  `[IO.File]::WriteAllText($path, $text, [System.Text.UTF8Encoding]::new($false))`, or explicit
  Unicode-pair literals (`[char]0xD83D + [char]0xDCCA` for 📊). The validator's `U+FFFD` check
  catches the worst case (mojibake) but cannot detect characters silently stripped to nothing.
- **Never use regex to bulk-edit balanced HTML.** This skill has burned twice on regex strip scripts
  that ate matched-pair `</div>` closes, producing inception-style nested-callout bugs that take a
  depth-tracking script to find. Make targeted, single-occurrence string replacements (with explicit
  before/after context) or rewrite the affected block end-to-end.
- **No shorthand in user-facing text.** All UI text — callouts, table headers, KPI labels, verdicts,
  badges — says `devices` and `requests`, never `devs` / `reqs`. Internal variable, column, and file
  names can stay short.
- **A moved metric is a question, not a verdict.** Never publish a regression verdict without the
  app's diagnostic ladder having been walked (Broker: Originator pre-check + dim slicing;
  Authenticator: volume floor + rate normalisation + error-reason decomposition).
- **Filename collision rule.** If a report already exists for the same end-date, do not silently
  overwrite. Open it, list its top-3 findings, and explicitly state in chat what changed in the new
  data before regenerating. A second run on the same window without a delta is wasted work.
- **Do not create a separate Markdown summary** of either report — the HTML *is* the deliverable.
- **Do not commit** any report. They live in `$env:USERPROFILE\android-oce-reports\` precisely so
  they can't be staged accidentally.

---

## Router checklist

- [ ] Mode resolved (`both` default) and stated in chat before work starts.
- [ ] Window resolved **once** and the same `-EndDate` passed to every bootstrap call.
- [ ] Each selected playbook read **in full** by the agent that owns it.
- [ ] In `both` mode, the two playbooks ran as **parallel** sub-agents, not interleaved in one context.
- [ ] Each report passed its own `validate-report.ps1 -App <app>` run.
- [ ] `oce-index-<curEnd>.html` built (in `both` mode) **after** both validators passed.
- [ ] File paths reported in chat; no report content pasted into chat; nothing committed.