ML Pipeline Health Report
Produce an actionable report on Metaflow ML pipeline runs using Metaflow MCP tools and the debug-run skill.
Configuration
See _shared/report-config.md for the standard config resolution pattern.
Config schema (~/.claude/skills/ml-pipeline-report/config.json):
flows: list of Metaflow flow name stringsnamespaces: list of Metaflow namespace strings (default: ["production"])lookback_days: number (default: 7)
Required scope: at least one of flows or namespaces.
Arguments
- flows: list of Metaflow flow names to check
- namespaces: Metaflow namespaces to search (default:
["production"])
- lookback_days: how far back to check runs (default: match your cadence — 1 for daily, 7 for weekly)
Workflow
Discover recent runs. For each configured flow/namespace, query Metaflow for runs within lookback_days. Use the Metaflow MCP tools or invoke the debug-run skill pattern to inspect runs.
Classify run status. For each run: succeeded, failed, running, or timed out. Record run ID, start time, end time (if complete), and duration.
Diagnose failures. For every failed run, get the failing step, error message, and stack trace summary. Use Skill(skill="debug-run", args="<FlowName>/<run_id>") to inspect the run's step-level status and logs.
Check resource usage. For running flows, note duration so far. Flag any run exceeding 2x its typical duration as "long-running" (compare against recent successful run durations if available).
Generate report. Output markdown:
## ML Pipeline Health Report — {date}
Period: last {lookback_days} days | Flows checked: {N} | Total runs: {N}
### Summary
- Failed: {count} | Running: {count} | Succeeded: {count} | Timed out: {count}
### Failed Runs
(For each: flow name, run ID, failing step, error summary, duration, link/command to inspect)
### Long-Running
(For each: flow name, run ID, duration so far, expected duration)
### Currently Running
(Compact list: flow name, run ID, start time, current step)
### Recently Succeeded
(Compact table: flow name, run ID, duration, completion time)
Deliver as HTML. Follow the shared HTML delivery pattern in _shared/html-delivery.md. Report name: ml-pipeline. TLDR includes flows checked, failure count, and longest running flow.
Terminate. Report is complete when all sections are populated.
Design Principles
- Team-level only. Aggregate to team level — it is the right granularity for a periodic digest. Individual-level detail (commits per person, PR count per author) is too noisy for a team report.
- Deterministic math first, LLM narrates only. All classification (severity, flaky vs broken, pass/fail) must be computed deterministically from data. The LLM writes prose around the numbers but never assigns severity or makes classification judgments.
- Pair metrics with counter-metrics. Never report velocity without stability. If showing deploy frequency, also show change failure rate. If showing PR throughput, also show rework rate.
Golden Rules
- Always get the failing step for failed runs. "Run X failed" without the step name and error is useless. Drill into step-level status.
- Check ALL configured flows, not a sample. Every flow in scope gets checked.
- Include run IDs. Every run reference includes the run ID so the reader can
debug-run it directly.
- Failed runs first. Report sections ordered by severity.
- Duration context matters. A 2-hour run is fine if it usually takes 2 hours. Flag runs that are significantly longer than their historical average.
- "All clear" is still a report. Full structure with run counts and completion times even when everything passed.
Anti-Rationalization Counter-Table
| Excuse |
Reality |
| "Run failed but I didn't check which step." |
The step name and error message are what the ML engineer needs. Get them. |
| "I checked the main flow, the others are probably fine." |
Check ALL configured flows. A failed feature pipeline can block downstream training. |
| "The run is still going, I'll skip it." |
Running flows need duration tracking. A stuck run is worse than a failed one — at least failures are visible. |
| "I don't have the typical duration to compare against." |
Use recent successful runs from the same lookback period as a baseline. No historical data = note it, don't skip the check. |
Termination Labels
| Label |
Meaning |
report_complete |
All flows checked, failures have step-level detail and error messages |
report_partial |
Some flows checked but errors prevented full coverage |
no_flows_found |
No runs found for configured flows — scope may be wrong |
api_error |
Metaflow API unreachable or returning errors |
Self-Review Checklist
1---2name: ml-pipeline-report3description: Use when checking ML pipeline health, Metaflow flow status, failed runs, or generating an ML pipeline report. Trigger phrases: metaflow status, ML pipeline health, flow runs, metaflow report, are my flows running, ML pipeline check.4---56# ML Pipeline Health Report78Produce an actionable report on Metaflow ML pipeline runs using Metaflow MCP tools and the debug-run skill.910## Configuration1112See [`_shared/report-config.md`](../_shared/report-config.md) for the standard config resolution pattern.1314**Config schema** (`~/.claude/skills/ml-pipeline-report/config.json`):15- `flows`: list of Metaflow flow name strings16- `namespaces`: list of Metaflow namespace strings (default: `["production"]`)17- `lookback_days`: number (default: 7)1819**Required scope:** at least one of `flows` or `namespaces`.2021## Arguments2223- **flows**: list of Metaflow flow names to check24- **namespaces**: Metaflow namespaces to search (default: `["production"]`)25- **lookback_days**: how far back to check runs (default: match your cadence — 1 for daily, 7 for weekly)2627## Workflow28291. **Discover recent runs.** For each configured flow/namespace, query Metaflow for runs within lookback_days. Use the Metaflow MCP tools or invoke the `debug-run` skill pattern to inspect runs.30312. **Classify run status.** For each run: succeeded, failed, running, or timed out. Record run ID, start time, end time (if complete), and duration.32333. **Diagnose failures.** For every failed run, get the failing step, error message, and stack trace summary. Use `Skill(skill="debug-run", args="<FlowName>/<run_id>")` to inspect the run's step-level status and logs.34354. **Check resource usage.** For running flows, note duration so far. Flag any run exceeding 2x its typical duration as "long-running" (compare against recent successful run durations if available).36375. **Generate report.** Output markdown:3839```40## ML Pipeline Health Report — {date}41Period: last {lookback_days} days | Flows checked: {N} | Total runs: {N}4243### Summary44- Failed: {count} | Running: {count} | Succeeded: {count} | Timed out: {count}4546### Failed Runs47(For each: flow name, run ID, failing step, error summary, duration, link/command to inspect)4849### Long-Running50(For each: flow name, run ID, duration so far, expected duration)5152### Currently Running53(Compact list: flow name, run ID, start time, current step)5455### Recently Succeeded56(Compact table: flow name, run ID, duration, completion time)57```58596. **Deliver as HTML.** Follow the shared HTML delivery pattern in [`_shared/html-delivery.md`](../_shared/html-delivery.md). Report name: `ml-pipeline`. TLDR includes flows checked, failure count, and longest running flow.60617. **Terminate.** Report is complete when all sections are populated.6263## Design Principles64651. **Team-level only.** Aggregate to team level — it is the right granularity for a periodic digest. Individual-level detail (commits per person, PR count per author) is too noisy for a team report.662. **Deterministic math first, LLM narrates only.** All classification (severity, flaky vs broken, pass/fail) must be computed deterministically from data. The LLM writes prose around the numbers but never assigns severity or makes classification judgments.673. **Pair metrics with counter-metrics.** Never report velocity without stability. If showing deploy frequency, also show change failure rate. If showing PR throughput, also show rework rate.6869## Golden Rules70711. **Always get the failing step for failed runs.** "Run X failed" without the step name and error is useless. Drill into step-level status.722. **Check ALL configured flows, not a sample.** Every flow in scope gets checked.733. **Include run IDs.** Every run reference includes the run ID so the reader can `debug-run` it directly.744. **Failed runs first.** Report sections ordered by severity.755. **Duration context matters.** A 2-hour run is fine if it usually takes 2 hours. Flag runs that are significantly longer than their historical average.766. **"All clear" is still a report.** Full structure with run counts and completion times even when everything passed.7778## Anti-Rationalization Counter-Table7980| Excuse | Reality |81|---|---|82| "Run failed but I didn't check which step." | The step name and error message are what the ML engineer needs. Get them. |83| "I checked the main flow, the others are probably fine." | Check ALL configured flows. A failed feature pipeline can block downstream training. |84| "The run is still going, I'll skip it." | Running flows need duration tracking. A stuck run is worse than a failed one — at least failures are visible. |85| "I don't have the typical duration to compare against." | Use recent successful runs from the same lookback period as a baseline. No historical data = note it, don't skip the check. |8687## Termination Labels8889| Label | Meaning |90|---|---|91| `report_complete` | All flows checked, failures have step-level detail and error messages |92| `report_partial` | Some flows checked but errors prevented full coverage |93| `no_flows_found` | No runs found for configured flows — scope may be wrong |94| `api_error` | Metaflow API unreachable or returning errors |9596## Self-Review Checklist9798- [ ] All configured flows were checked99- [ ] Failed runs have failing step name and error message100- [ ] Run IDs present for every run referenced101- [ ] Long-running section compares against historical duration102- [ ] Failed section appears before succeeded section103- [ ] Report has date header and flow list noted104- [ ] HTML version uploaded to S3 with commuter link (unless `--no-html` or upload failed with noted fallback)105- [ ] Slack/chat delivery uses TLDR + link, not the full report
npow