Pipeline Incident Triage

Pipeline incident triage

Follow references/incident-runbook.md in order. Do not skip to a hypothesis.

Opening move

Ask exactly two things if they are not already clear, and only two:

1. Is the data stale, or is it wrong and already published?
2. What reads from it?

Stale is a delay. Wrong and published is an emergency, and the first action is to stop the next run, not to debug.

Gather the evidence yourself when you can

Read references/execution-model.md. During an incident the difference between asking the user to run six commands and reading the answer directly is the difference between a five minute triage and a thirty minute one.

With the MCP server, run these through execute_sql_readonly before asking the user anything:

1. Freshness and row count by partition on the suspect table. The partial load that reports no error shows up here and nowhere else.
2. get_table_info on the table and its upstreams, to see whether a schema moved under the pipeline.
3. Job history for the last 24 hours filtered to the tables involved, to find the job that actually failed rather than the task that reported it.

With the gcloud CLI, add the Airflow and Cloud Logging side, which the BigQuery MCP server cannot see.

Then ask the user only what you could not read: whether the numbers are wrong or merely late, and who consumes the table.

Commands for the parts you cannot read

# Which Airflow tasks failed in the last day
gcloud composer environments run ENV_NAME --location REGION \
  tasks states-for-dag-run -- DAG_ID RUN_ID

# The BigQuery job behind a failed task
bq --location=US show --format=prettyjson -j JOB_ID

# Errors across the project, last 2 hours
gcloud logging read \
  'severity>=ERROR AND timestamp>="'"$(date -u -d '2 hours ago' +%Y-%m-%dT%H:%M:%SZ)"'"' \
  --limit=50 --format='table(timestamp, resource.type, jsonPayload.message)'

Freshness check on the suspect table:

SELECT
  table_name,
  TIMESTAMP_MILLIS(last_modified_time) AS last_modified,
  TIMESTAMP_DIFF(CURRENT_TIMESTAMP(), TIMESTAMP_MILLIS(last_modified_time), MINUTE) AS minutes_stale,
  row_count
FROM `mart_retail.__TABLES__`
WHERE table_id = 'fct_order';

Row count by partition, which catches partial loads that no error reports:

SELECT
  partition_id,
  total_rows,
  TIMESTAMP_MILLIS(last_modified_time) AS last_modified
FROM `mart_retail`.INFORMATION_SCHEMA.PARTITIONS
WHERE table_name = 'fct_order'
ORDER BY partition_id DESC
LIMIT 14;

A partition with 12 percent of yesterday's rows and no error is the classic silent failure. Always look at this before declaring a pipeline healthy.

Cascades

In Dataform and Airflow, the first failure is the incident and everything after is noise. Find the earliest failed action by timestamp and debug only that one. Reporting fifteen failures when there is one root cause wastes the responder's attention.

Cost spike triage

SELECT
  user_email,
  ROUND(SUM(total_bytes_processed) / POW(1024, 4), 2) AS tb,
  COUNT(*) AS jobs
FROM `region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECT
WHERE creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 24 HOUR)
GROUP BY user_email
ORDER BY tb DESC
LIMIT 20;

Then compare against the same window a week earlier. A spike is only a spike relative to a baseline, and most reported spikes are a normal Monday.

Backfills

Never run an unbounded backfill. State the window, the estimated bytes, and the cost before running anything, then run one partition window at a time:

-- one day, into one partition
CREATE OR REPLACE TABLE `mart_retail.fct_order`
PARTITION BY DATE(ordered_at)
AS SELECT ... WHERE ordered_at >= '2026-03-01' AND ordered_at < '2026-03-02';

For a real backfill, prefer MERGE or a partition-scoped WRITE_TRUNCATE job over recreating the table, because recreating drops the description, the labels, and the partition filter requirement.

Output format

STATUS:   stale | wrong-and-published | degraded | resolved
SCOPE:    <what is affected, in business terms>
ROOT CAUSE: <one sentence>
EVIDENCE: <the specific log line or row count>
FIX:      <exact commands or SQL>
BACKFILL: <window, estimated bytes, or "none needed">
GUARDRAIL: <the assertion or check that would have caught this>

Rules

• Never guess at a root cause when you have not seen the error text. Ask for it.
• Never suggest a change to production data without saying what it overwrites.
• If the evidence supports two causes, say both and say which check separates them. A confident wrong diagnosis costs more than an honest fork.