Connect Debugging

Redpanda Connect: Debugging

Redpanda Connect (formerly Benthos) provides built-in tools for validating configs before deployment, tuning structured logs, exporting metrics to Prometheus or StatsD, shipping traces to OpenTelemetry, and probing pipeline health over HTTP. This skill covers the full debugging lifecycle: lint before you deploy, tune logging to see what is happening, scrape metrics to quantify it, and use the health endpoints to build reliable probes.

The HTTP server (default 0.0.0.0:4195) provides /ping (liveness), /ready (readiness — 200 only when both input and output are connected), and /metrics//stats (metrics). Use debug_endpoints: true to expose pprof.

Quickstart

# 1. Lint a config (exit 1 on any error)
rpk connect lint ./pipeline.yaml

# Lint with extra checks: flag deprecated fields, require labels
rpk connect lint --deprecated --labels ./pipeline.yaml

# Lint without failing on unset env vars (useful in CI without secrets)
rpk connect lint --skip-env-var-check ./pipeline.yaml

# 2. Dry-run: lint + test actual connections (does the broker accept the creds?)
rpk connect dry-run ./pipeline.yaml

# Dry-run with verbose per-file output
rpk connect dry-run --verbose ./pipeline.yaml

# 3. Run with DEBUG logging to see what is happening
rpk connect run --log.level DEBUG ./pipeline.yaml

# 4. Check health endpoints (pipeline must be running on port 4195)
curl -s http://localhost:4195/ping          # 200 OK = process alive
curl -s -o /dev/null -w "%{http_code}" \
  http://localhost:4195/ready              # 200 = input+output connected, 503 = not yet

# 5. Scrape metrics (when metrics: prometheus or json_api is configured)
curl -s http://localhost:4195/metrics      # Prometheus text format
curl -s http://localhost:4195/stats        # Alias; also serves json_api format

Add this block to a pipeline config to expose Prometheus metrics and bump logging to DEBUG for initial troubleshooting:

http:
  address: 0.0.0.0:4195
  debug_endpoints: false   # set true to enable /debug/pprof/* endpoints

logger:
  level: DEBUG             # OFF FATAL ERROR WARN INFO DEBUG TRACE ALL NONE
  format: logfmt           # logfmt or json
  add_timestamp: true
  static_fields:
    '@service': redpanda-connect

metrics:
  prometheus: {}
  mapping: ""              # optional Bloblang filter/rename for metric names

Linting and Validation

rpk connect lint parses pipeline YAML and reports structural errors before the pipeline ever runs. It exits with status code 1 if any lint error is found.

Flags

Note: internal/cli/custom_lint.go implements customLintCli(), which is the mcp-server lint subcommand (directory-only). The top-level rpk connect lint accepting file paths/globs is from the upstream Benthos framework. The flags below apply to both.

# Lint a single file
rpk connect lint pipeline.yaml

# Lint all YAML files in a directory tree
rpk connect lint ./config/...

# CI-friendly: strict mode, skip unset env vars, verbose output
rpk connect lint --deprecated --labels --skip-env-var-check --verbose ./pipelines/

Dry-run (connection test)

rpk connect dry-run goes beyond lint: it parses the config and attempts to establish actual connections to every component. It exits 1 if any connection test fails. This is the fastest way to detect credential/network problems before deploying. It accepts an optional --redpanda-license flag for testing Enterprise components.

rpk connect dry-run ./pipeline.yaml
rpk connect dry-run --verbose ./pipeline.yaml

Logger Configuration

Configure the logger: section to control level and format. Full schema with all fields in references/logging-metrics-tracing.md.

Key levels (grounded in docs/modules/components/pages/logger/about.adoc): OFF, FATAL, ERROR, WARN, INFO (default), DEBUG, TRACE, ALL, NONE.

logger:
  level: INFO     # OFF FATAL ERROR WARN INFO DEBUG TRACE ALL NONE
  format: logfmt  # logfmt (default) or json
  add_timestamp: true
  static_fields:
    '@service': redpanda-connect

Override level on the CLI without editing the config:

rpk connect run --log.level DEBUG ./pipeline.yaml

Metrics Backends

Configure the metrics: section. All backends accept mapping: to filter or rename metric paths. Full YAML schemas for all backends in references/logging-metrics-tracing.md.

Key standard metric names: input_received, input_latency, output_sent.

Tracing (OpenTelemetry)

Send distributed traces to an OpenTelemetry collector. Full schema in references/logging-metrics-tracing.md.

tracer:
  open_telemetry_collector:
    service: redpanda-connect
    grpc:
      - address: "localhost:4317"
        secure: false
    sampling:
      enabled: true
      ratio: 0.1    # sample 10% for high-volume pipelines

Health Endpoints

The HTTP server (default 0.0.0.0:4195) exposes these endpoints. Grounded in docs/modules/components/pages/http/about.adoc.

/ready returning 503 is normal at startup. In Kubernetes use /ping as liveness and /ready as readiness.

Set http.debug_endpoints: true to expose /debug/pprof/* (CPU profile, heap, goroutine, block, mutex) and /debug/config/json|yaml. See full endpoint list and curl examples in references/logging-metrics-tracing.md.

Enterprise License and Enterprise Features

Several Redpanda Connect features are Enterprise and require a valid Redpanda Enterprise license: all enterprise connectors (including the CDC inputs), connector allow/deny lists, secrets management (remote runtime lookup), the redpanda: configuration service block, and FIPS compliance. After the 30-day trial expires you are blocked from using enterprise connectors until you upgrade. Full per-feature config keys (including the logminer{} sub-block, the connector_list.yaml allow/deny structure, and --secrets URN schemes) are in references/enterprise-features.md.

Enterprise components (CDC inputs: postgres_cdc, mysql_cdc, mongodb_cdc, oracledb_cdc, microsoft_sql_server_cdc, aws_dynamodb_cdc, gcp_spanner_cdc, salesforce_cdc) require a Redpanda Enterprise license. Without one, these components fail at connection time with a license error. (tigerbeetle_cdc is the exception among CDC inputs: it is a certified community component — no license required — but it needs a CGO-enabled Connect build; it is absent from rpk connect and the standard Docker image, so a "component not found" error there is a build issue, not a license issue.)

Default license path: /etc/redpanda/redpanda.license (grounded in internal/license/service.go, constant defaultLicenseFilepath)

Environment variables:

• REDPANDA_LICENSE — inline license string
• REDPANDA_LICENSE_FILEPATH — path to license file

CLI flag:

rpk connect run --redpanda-license "$(cat redpanda.license)" ./pipeline.yaml
rpk connect dry-run --redpanda-license "$(cat redpanda.license)" ./pipeline.yaml

The license service logs on startup:

level=info msg="Successfully loaded Redpanda license" license_org=MyOrg license_type=enterprise expires_at=2026-01-01T00:00:00Z

If no valid Enterprise license is found, a 10-year open-source license is applied automatically. Attempting to use an Enterprise component under the open-source license produces an error at connection time (visible in dry-run).

Allow/deny list: /etc/redpanda/connector_list.yaml restricts which components an instance may run (allow: OR deny:, never both). If a component unexpectedly "does not exist," check this file. See references/enterprise-features.md.

Secrets management: the --secrets <urn> flag resolves ${SECRET} interpolations from a remote system at runtime. Schemes: env:, none:, aws:, gcp:, az:, redis:. URNs are tried in order; first hit wins.

Common Failure Modes

See references/failure-modes.md for a step-by-step triage checklist. Quick index:

Reference Directory

• lint-and-validate.md: rpk connect lint and rpk connect dry-run flags, exit codes, what each catches, env-var interpolation handling, and CI integration patterns.
• logging-metrics-tracing.md: Full logger config (all levels and fields), all metrics backends (prometheus/statsd/json_api/logger) with complete YAML, OpenTelemetry tracer config, and the /ping//ready endpoint reference.
• failure-modes.md: Triage checklist for common Connect failure modes: connection/TLS/auth errors, checkpoint/cache failures, Enterprise license errors, backpressure and consumer lag, and memory pressure — each with the exact knob to turn.
• enterprise-features.md: Redpanda Connect enterprise features and their nested config keys (all require an Enterprise license): license loading order (--redpanda-license, REDPANDA_LICENSE, REDPANDA_LICENSE_FILEPATH, /etc/redpanda/redpanda.license); enterprise CDC inputs with full skeletons (postgres_cdc slot/heartbeat, mysql_cdc binlog/checkpoint_cache, mongodb_cdc change streams, oracledb_cdc logminer{} block); connector allow/deny lists (/etc/redpanda/connector_list.yaml); secrets management --secrets URN schemes; the redpanda: configuration service block; and FIPS compliance.