# Pipeline Incident Triage

> Triage a failed or wrong data pipeline on GCP and produce a fix plus a postmortem. Use when the user says a pipeline failed, a DAG is red, a table is stale, numbers look wrong, an assertion failed, a dashboard is broken, a job is stuck, or costs spiked unexpectedly. Also use when the user pastes an error from BigQuery, Airflow, Dataform, or Cloud Run.

- Skill: `rk-chavali/pipeline-incident-triage` (Agent Skill)
- Install (CLI): `npx skillmds@latest add rk-chavali/pipeline-incident-triage`
- Raw SKILL.md: https://api.skillmd.com/api/skills/rk-chavali/pipeline-incident-triage/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: rk-chavali (https://skillmd.com/u/rk-chavali)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/rk-chavali/pipeline-incident-triage

---


# Pipeline incident triage

Follow `references/incident-runbook.md` in order. Do not skip to a hypothesis.

## Opening move

Ask exactly two things if they are not already clear, and only two:

1. Is the data stale, or is it wrong and already published?
2. What reads from it?

Stale is a delay. Wrong and published is an emergency, and the first action is to
stop the next run, not to debug.

## Gather the evidence yourself when you can

Read `references/execution-model.md`. During an incident the difference between
asking the user to run six commands and reading the answer directly is the
difference between a five minute triage and a thirty minute one.

**With the MCP server**, run these through `execute_sql_readonly` before asking
the user anything:

1. Freshness and row count by partition on the suspect table. The partial load
   that reports no error shows up here and nowhere else.
2. `get_table_info` on the table and its upstreams, to see whether a schema
   moved under the pipeline.
3. Job history for the last 24 hours filtered to the tables involved, to find
   the job that actually failed rather than the task that reported it.

**With the `gcloud` CLI**, add the Airflow and Cloud Logging side, which the
BigQuery MCP server cannot see.

Then ask the user only what you could not read: whether the numbers are wrong or
merely late, and who consumes the table.

## Commands for the parts you cannot read

```bash
# Which Airflow tasks failed in the last day
gcloud composer environments run ENV_NAME --location REGION \
  tasks states-for-dag-run -- DAG_ID RUN_ID

# The BigQuery job behind a failed task
bq --location=US show --format=prettyjson -j JOB_ID

# Errors across the project, last 2 hours
gcloud logging read \
  'severity>=ERROR AND timestamp>="'"$(date -u -d '2 hours ago' +%Y-%m-%dT%H:%M:%SZ)"'"' \
  --limit=50 --format='table(timestamp, resource.type, jsonPayload.message)'
```

Freshness check on the suspect table:

```sql
SELECT
  table_name,
  TIMESTAMP_MILLIS(last_modified_time) AS last_modified,
  TIMESTAMP_DIFF(CURRENT_TIMESTAMP(), TIMESTAMP_MILLIS(last_modified_time), MINUTE) AS minutes_stale,
  row_count
FROM `mart_retail.__TABLES__`
WHERE table_id = 'fct_order';
```

Row count by partition, which catches partial loads that no error reports:

```sql
SELECT
  partition_id,
  total_rows,
  TIMESTAMP_MILLIS(last_modified_time) AS last_modified
FROM `mart_retail`.INFORMATION_SCHEMA.PARTITIONS
WHERE table_name = 'fct_order'
ORDER BY partition_id DESC
LIMIT 14;
```

A partition with 12 percent of yesterday's rows and no error is the classic
silent failure. Always look at this before declaring a pipeline healthy.

## Cascades

In Dataform and Airflow, the first failure is the incident and everything after
is noise. Find the earliest failed action by timestamp and debug only that one.
Reporting fifteen failures when there is one root cause wastes the responder's
attention.

## Cost spike triage

```sql
SELECT
  user_email,
  ROUND(SUM(total_bytes_processed) / POW(1024, 4), 2) AS tb,
  COUNT(*) AS jobs
FROM `region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECT
WHERE creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 24 HOUR)
GROUP BY user_email
ORDER BY tb DESC
LIMIT 20;
```

Then compare against the same window a week earlier. A spike is only a spike
relative to a baseline, and most reported spikes are a normal Monday.

## Backfills

Never run an unbounded backfill. State the window, the estimated bytes, and the
cost before running anything, then run one partition window at a time:

```sql
-- one day, into one partition
CREATE OR REPLACE TABLE `mart_retail.fct_order`
PARTITION BY DATE(ordered_at)
AS SELECT ... WHERE ordered_at >= '2026-03-01' AND ordered_at < '2026-03-02';
```

For a real backfill, prefer `MERGE` or a partition-scoped `WRITE_TRUNCATE` job
over recreating the table, because recreating drops the description, the labels,
and the partition filter requirement.

## Output format

```
STATUS:   stale | wrong-and-published | degraded | resolved
SCOPE:    <what is affected, in business terms>
ROOT CAUSE: <one sentence>
EVIDENCE: <the specific log line or row count>
FIX:      <exact commands or SQL>
BACKFILL: <window, estimated bytes, or "none needed">
GUARDRAIL: <the assertion or check that would have caught this>
```

## Rules

- Never guess at a root cause when you have not seen the error text. Ask for it.
- Never suggest a change to production data without saying what it overwrites.
- If the evidence supports two causes, say both and say which check separates
  them. A confident wrong diagnosis costs more than an honest fork.