Estuary Task Health

estuary-task-health - Single Task Health Check

Run four checks against a single task and synthesize a verdict. Covers control-plane state, data flow, recent errors, and recent spec changes.

Applies to: captures, materializations, derivations

Step 0: Get the Task Name

If the user hasn't provided a full task name, find it:

flowctl catalog list --prefix <tenant>/ | grep -E "capture|materialize|derivation"

Step 1: Control-Plane Status

Is the task running at all?

flowctl catalog status <task> -o json | jq '{
  status:             .status.type,
  summary:            .status.summary,
  recent_failures:    .status.controller.activation.recentFailureCount,
  last_failure_error: (.status.controller.activation.lastFailure.fields.error // .status.controller.activation.lastFailure.message // null)
}'

Note: last_failure_error shows the raw error string, which is much more readable than the full lastFailure object. For schema validation failures the full object includes the entire failing document — use .fields.error to avoid walls of JSON.

For full status field reference, see the estuary-catalog-status skill.

Step 2: Data Flow

Is data actually moving? (A task can show OK while stalled.)

For materializations:

flowctl raw stats --task <task> --since 24h | \
  jq -s '{
    total_gb:     ([.[] | select(.materialize != null) | .materialize | to_entries[] | .value.right.bytesTotal // 0] | add // 0 | . / 1073741824 * 10 | round / 10),
    total_docs:   ([.[] | select(.materialize != null) | .materialize | to_entries[] | .value.right.docsTotal  // 0] | add // 0),
    transactions: ([.[] | select(.materialize != null)] | length)
  }'

For captures:

flowctl raw stats --task <task> --since 24h | \
  jq -s '{
    total_gb:     ([.[] | select(.capture != null) | .capture | to_entries[] | .value.out.bytesTotal // 0] | add // 0 | . / 1073741824 * 10 | round / 10),
    total_docs:   ([.[] | select(.capture != null) | .capture | to_entries[] | .value.out.docsTotal  // 0] | add // 0),
    transactions: ([.[] | select(.capture != null)] | length)
  }'

transactions: 0 with OK status = task is running but no data is moving. Common causes: quiet source, sync schedule delay (default 30 min for materializations), connector in retry backoff.

Note: Push-mode connectors (Kafka, webhooks) produce one transaction per message — high transaction counts (100k+/day) are normal and expected.

For hourly breakdown (spot gaps and batch patterns), see the estuary-task-stats skill.

Step 3: Recent Errors and Warnings

Any error patterns in the last 24h?

flowctl logs --task <task> --since 24h | jq -c 'select(.level == "error" or .level == "warn") | {ts, level, message, error: .fields.error}'

Look for repeated errors — a single error followed by recovery is normal. Errors repeating every few minutes indicate the task is stuck. Warnings without errors often signal transient issues.

If errors are found, search Kapa for tribal knowledge on the specific error message (if the Estuary MCP is configured): Search kapa ai knowledge sources for "<error message>"

Not configured? Set it up: https://docs.estuary.dev/features/mcp-integration/

Also check the connector's documentation URL for known issues — find it with:

flowctl raw get --table connector_tags --query 'documentation_url=ilike.*<connector-name>*' --query 'select=documentation_url' --output yaml

Benign warnings to ignore:

• "could not fetch queued overload time" — Snowflake internal metric, harmless
• "Collection not available" / "Collection binding exists but has no journals available" — dekaf/push connector startup, harmless

For full log analysis patterns, see the estuary-logs skill.

Step 4: Recent Spec Changes

Did something change recently that might explain the issue?

flowctl catalog history --name <task> -o json | \
  jq -s '[.[:5][] | {at: .publication.publishedAt, who: .publication.userEmail, detail: .publication.detail}]'

Look for recent manual publishes, auto-discover events, or dependency changes that correlate with when the issue started.

For full publication history analysis, see the estuary-catalog-history skill.

Synthesis

After running all four checks, summarize:

Verdict:

• Healthy — OK status, data flowing, no errors, no recent changes
• Stalled — OK status, zero stats, no errors → sync schedule delay or quiet source
• Degraded — WARNING status, intermittent errors, recovering on its own
• Failing — ERROR status or repeated errors not recovering → needs intervention
• Misconfigured — errors correlate with a recent spec change

Common error patterns and fixes:

Next steps by verdict:

• Stalled: check sync schedule, verify source has new data, check logs for retry backoff
• Degraded: monitor recentFailureCount; if climbing, treat as Failing
• Failing: search Kapa for the error, fix config and re-publish, or use estuary-connector-restart to reset; contact support if logs show a connector panic
• Misconfigured: use estuary-catalog-history to diff the spec change; re-publish with fix

Optional: Check Connected Tasks

After completing the health check, offer to check the rest of the pipeline. A single capture may feed many materializations, and a single materialization may read from many captures.

If the checked task is a materialization, --connected finds upstream captures directly:

flowctl catalog status <materialization> --connected -o json | \
  jq -c 'select(.catalogType == "capture" or .catalogType == "materialization") | {name: .catalogName, type: .catalogType, status: .status.type}'

If the checked task is a capture, use a two-hop approach via a collection — --connected on a collection finds both directions:

# Step 1: get any collection produced by the capture
COLLECTION=$(flowctl catalog status <capture> --connected -o json 2>/dev/null | \
  jq -r 'select(.catalogType == "collection") | .catalogName' | head -1)

# Step 2: --connected on the collection shows both the capture and downstream materializations
flowctl catalog status "$COLLECTION" --connected -o json | \
  jq -c 'select(.catalogType == "capture" or .catalogType == "materialization") | {name: .catalogName, type: .catalogType, status: .status.type}'

Important: head -1 only checks one collection. Different collections from the same capture can feed different materializations. Show the user what was found and ask:

1. Do these look like the right connected tasks, or should all collections be checked for additional materializations?
2. Which connected tasks would they like to health-check?

To check all collections for completeness (may be slow for large captures):

flowctl catalog status <capture> --connected -o json 2>/dev/null | \
  jq -r 'select(.catalogType == "collection") | .catalogName' | \
  while read col; do
    flowctl catalog status "$col" --connected -o json 2>/dev/null | \
      jq -c 'select(.catalogType == "materialization") | .catalogName'
  done | sort -u

Ask the user if they'd like to run the same health check on any of the connected tasks.

Related Skills

• estuary-catalog-status — Deep dive into control-plane status fields
• estuary-task-stats — Hourly breakdown and per-collection stats
• estuary-logs — Full log analysis with filtering patterns
• estuary-catalog-history — Publication timeline and spec diffs
• estuary-connector-restart — Pause and restart a task via flowctl