Estuary Logs

flowctl logs - Search and analyze Estuary task logs

Concepts: Logs contain structured JSON with fields: ts (timestamp), level, message, and optional fields (extra context). Tasks are captures, materializations, or derivations running in Estuary.

Prerequisites

flowctl must already be authenticated, as per estuary-flowctl-setup skill.

Finding your task name: If you don't already know the task name, use flowctl catalog list --prefix <tenant/prefix> to narrow it down, or copy it from the Captures or Materializations page in the Estuary web app. It looks like org/prefix/connector-name, for example:

acmeCo/ops/source-postgres

Basic Usage

Quick check — pipe directly to the terminal:

flowctl logs --task <task> --since 1h | tail | jq -c

For investigations, save to a file first (see Effective Log Analysis Workflow).

Live streaming (follow mode)

If a task is currently running and you want logs as they appear:

flowctl logs --task <task> --follow

Combine with --since to get recent history plus live tailing:

flowctl logs --task <task> --since 1h --follow

Choosing --since Values

Note on --since accuracy: The actual start time snaps back to the previous log fragment boundary, so you may get somewhat more history than requested (e.g. --since 1h may return up to several hours). This is expected, so filter downstream to find specific time ranges

Always use --since unless you need logs since task creation (e.g. for initial DDL statements). Log history can span 1+ years for long-running tasks and will be very slow to fetch without a time bound.

Effective Log Analysis Workflow

Recommended approach for investigating issues:

# 1. Save relevant logs to file (allows re-processing without re-fetching)
flowctl logs --task <task> --since 2d > task.logs

# 2. Extract useful fields, removing noise (.shard, ._meta clutter the output)
jq -c 'del(.shard, ._meta) | {ts, message} + .' task.logs

# 3. Filter out repeated irrelevant messages
jq -c 'del(.shard, ._meta) | {ts, message} + .' task.logs | grep -v "materialization progress"

Why this pattern:

• del(.shard, ._meta) removes verbose internal metadata that rarely helps debugging
• {ts, message} + . puts timestamp and message first while preserving other fields
• Saving to file lets you iterate on filters without re-fetching logs
• grep -v removes noisy repeated messages (e.g., periodic progress lines)

Quick Timeline (Minimal Output)

When you just need timestamps and messages:

flowctl logs --task <task> --since 1d | jq -c '{ts, message}'

Filtering for Issues

Filter for errors and warnings only:

flowctl logs --task <task> --since 1d | jq 'select(.level == "error" or .level == "warn")'

Get timeline with error detail:

flowctl logs --task <task> --since 1d | jq -c '{ts, message, level, error: .fields.error}'

Quick Grep Patterns

# Errors and failures
flowctl logs --task <task> --since 72h | jq -c 'del(.shard, ._meta) | {ts, message} + .' | grep -iE "shard failed|error|fail|warning"

# Task restarts (each new connector run)
flowctl logs --task <task> --since 72h | jq -c 'del(.shard, ._meta) | {ts, message} + .' | grep -iE "initialized catalog task term|started connector container"

# Transaction timing (Snowflake sync schedule)
flowctl logs --task <task> --since 72h | jq -c 'del(.shard, ._meta) | {ts, message} + .' | grep -E "nextScheduledAck|delaying|finished waiting"

# Retry storms (BigQuery DML quota issues)
flowctl logs --task <task> --since 72h | jq -c 'del(.shard, ._meta) | {ts, message} + .' | grep "previously submitted job failed, will retry"

Analyzing Repeated Failures

1. Count failure frequency by day:

flowctl logs --task <task> --since 7d | \
  jq -r 'select(.message == "shard failed") | .ts' | \
  awk -F'T' '{print $1}' | sort | uniq -c

2. Get root cause from shard failures:

flowctl logs --task <task> --since 1d | \
  jq -r 'select(.message == "shard failed") | {ts, error: .fields.error} | @json'

3. Find surrounding log entries around failures:

flowctl logs --task <task> --since 1d | \
  jq -c '{ts, message, level}' | grep -B2 "shard failed" | head -20

Understanding Common Log Messages

These messages are frequently misread as problems — here's what they actually mean:

Common Error Patterns

Materialization Transaction Failures

S3 Chunk Fetch Timeouts:

transactor.Load: querying Loads: chunk fetch: copying chunk stream:
decoding row: read tcp [IP]:[PORT]: read: connection timed out

• Occurs during transaction finalization when reading staged Load responses from S3
• Common with large materializations (100GB+ commits)

Snowflake PUT Operation Timeouts:

transactor.StartCommit: flush putFile: putWorker PUT to stage:
Post "https://[snowflake-host]:443/...": dial tcp: i/o timeout

• Occurs when uploading staged data to Snowflake
• Check for retry attempts: look for putWorker retrying PUT to stage messages

Getting More Verbose Logs

If default (info) logs don't show enough detail, you can increase the log level to debug. Add a shards stanza to your existing task spec alongside endpoint, then re-publish:

# Add this at the same level as your existing 'endpoint' block:
shards:
  logLevel: debug

Available levels (least → most verbose): error → warn → info (default) → debug

Remember to remove the logLevel override when done — debug produces a lot of output.

Related Skills

• estuary-catalog-status — Check whether a task is running, disabled, or failed before diving into logs
• estuary-connector-restart — If the connector needs to be paused and restarted via flowctl
• estuary-ssh-tunnels — If logs show SSH tunnel failures, connection timeouts, or permission denied errors

Tips

• flowctl logs --help lists all available flags
• Use jq -c for compact, line-by-line output (easier to grep)
• Look for shard failed messages — they include .fields.error with the root cause