connect-debugging

name

connect-debugging

description

Diagnoses, validates, and monitors Redpanda Connect pipelines. Use when a Redpanda Connect pipeline fails, stalls, drops messages, or won't start; when linting a config with rpk connect lint; tuning log level or format; reading Connect metrics (Prometheus, statsd, json_api) or traces (OpenTelemetry); checking /ready and /ping health endpoints; running a dry-run connection test; or diagnosing connector failures including auth errors, TLS errors, Enterprise license errors, backpressure, and checkpoint/cache issues. Covers Redpanda Connect enterprise features and their config keys: enterprise connectors including all CDC inputs (postgres_cdc, mysql_cdc, mongodb_cdc, oracledb_cdc logminer block), connector allow/deny lists, secrets management URNs, the redpanda: configuration service block, and FIPS compliance — all requiring an Enterprise license.

Redpanda Connect: Debugging

Redpanda Connect (formerly Benthos) provides built-in tools for validating configs before deployment, tuning structured logs, exporting metrics to Prometheus or StatsD, shipping traces to OpenTelemetry, and probing pipeline health over HTTP. This skill covers the full debugging lifecycle: lint before you deploy, tune logging to see what is happening, scrape metrics to quantify it, and use the health endpoints to build reliable probes.

The HTTP server (default 0.0.0.0:4195) provides /ping (liveness), /ready (readiness — 200 only when both input and output are connected), and /metrics//stats (metrics). Use debug_endpoints: true to expose pprof.

Quickstart

# 1. Lint a config (exit 1 on any error)
rpk connect lint ./pipeline.yaml

# Lint with extra checks: flag deprecated fields, require labels
rpk connect lint --deprecated --labels ./pipeline.yaml

# Lint without failing on unset env vars (useful in CI without secrets)
rpk connect lint --skip-env-var-check ./pipeline.yaml

# 2. Dry-run: lint + test actual connections (does the broker accept the creds?)
rpk connect dry-run ./pipeline.yaml

# Dry-run with verbose per-file output
rpk connect dry-run --verbose ./pipeline.yaml

# 3. Run with DEBUG logging to see what is happening
rpk connect run --log.level DEBUG ./pipeline.yaml

# 4. Check health endpoints (pipeline must be running on port 4195)
curl -s http://localhost:4195/ping          # 200 OK = process alive
curl -s -o /dev/null -w "%{http_code}" \
  http://localhost:4195/ready              # 200 = input+output connected, 503 = not yet

# 5. Scrape metrics (when metrics: prometheus or json_api is configured)
curl -s http://localhost:4195/metrics      # Prometheus text format
curl -s http://localhost:4195/stats        # Alias; also serves json_api format

Add this block to a pipeline config to expose Prometheus metrics and bump logging to DEBUG for initial troubleshooting:

http:
  address: 0.0.0.0:4195
  debug_endpoints: false   # set true to enable /debug/pprof/* endpoints

logger:
  level: DEBUG             # OFF FATAL ERROR WARN INFO DEBUG TRACE ALL NONE
  format: logfmt           # logfmt or json
  add_timestamp: true
  static_fields:
    '@service': redpanda-connect

metrics:
  prometheus: {}
  mapping: ""              # optional Bloblang filter/rename for metric names

Linting and Validation

rpk connect lint parses pipeline YAML and reports structural errors before the pipeline ever runs. It exits with status code 1 if any lint error is found.

Flags

Note: internal/cli/custom_lint.go implements customLintCli(), which is the mcp-server lint subcommand (directory-only). The top-level rpk connect lint accepting file paths/globs is from the upstream Benthos framework. The flags below apply to both.

Flag	Default	Effect
--deprecated	false	Fail on use of deprecated fields
--labels	false	Fail when components are missing label:
--skip-env-var-check	false	Do not fail when ${ENV_VAR} interpolations have no default and the variable is not set
--verbose	false	Print OK/FAILED for every file scanned
--env-file / -e	(none)	Load a .env dotenv file before linting
--secrets	env:	Secret lookup URNs (e.g. env:, none:)

# Lint a single file
rpk connect lint pipeline.yaml

# Lint all YAML files in a directory tree
rpk connect lint ./config/...

# CI-friendly: strict mode, skip unset env vars, verbose output
rpk connect lint --deprecated --labels --skip-env-var-check --verbose ./pipelines/

Dry-run (connection test)

rpk connect dry-run goes beyond lint: it parses the config and attempts to establish actual connections to every component. It exits 1 if any connection test fails. This is the fastest way to detect credential/network problems before deploying. It accepts an optional --redpanda-license flag for testing Enterprise components.

rpk connect dry-run ./pipeline.yaml
rpk connect dry-run --verbose ./pipeline.yaml

Logger Configuration

Configure the logger: section to control level and format. Full schema with all fields in references/logging-metrics-tracing.md.

Key levels (grounded in docs/modules/components/pages/logger/about.adoc): OFF, FATAL, ERROR, WARN, INFO (default), DEBUG, TRACE, ALL, NONE.

logger:
  level: INFO     # OFF FATAL ERROR WARN INFO DEBUG TRACE ALL NONE
  format: logfmt  # logfmt (default) or json
  add_timestamp: true
  static_fields:
    '@service': redpanda-connect

Override level on the CLI without editing the config:

rpk connect run --log.level DEBUG ./pipeline.yaml

Metrics Backends

Configure the metrics: section. All backends accept mapping: to filter or rename metric paths. Full YAML schemas for all backends in references/logging-metrics-tracing.md.

Backend	Use case
prometheus:	Production scraping via /metrics
statsd:	Push to StatsD/DataDog/InfluxDB
json_api:	Quick inspection with curl/jq at /metrics
logger:	Emit as log lines (no HTTP server needed)

Key standard metric names: input_received, input_latency, output_sent.

Tracing (OpenTelemetry)

Send distributed traces to an OpenTelemetry collector. Full schema in references/logging-metrics-tracing.md.

tracer:
  open_telemetry_collector:
    service: redpanda-connect
    grpc:
      - address: "localhost:4317"
        secure: false
    sampling:
      enabled: true
      ratio: 0.1    # sample 10% for high-volume pipelines

Health Endpoints

The HTTP server (default 0.0.0.0:4195) exposes these endpoints. Grounded in docs/modules/components/pages/http/about.adoc.

Endpoint	Purpose
/ping	Liveness: always 200 while process is up
/ready	Readiness: 200 when input AND output connected; 503 otherwise
/metrics, /stats	Metrics (prometheus or json_api format)

/ready returning 503 is normal at startup. In Kubernetes use /ping as liveness and /ready as readiness.

Set http.debug_endpoints: true to expose /debug/pprof/* (CPU profile, heap, goroutine, block, mutex) and /debug/config/json|yaml. See full endpoint list and curl examples in references/logging-metrics-tracing.md.

Enterprise License and Enterprise Features

Several Redpanda Connect features are Enterprise and require a valid Redpanda Enterprise license: all enterprise connectors (including the CDC inputs), connector allow/deny lists, secrets management (remote runtime lookup), the redpanda: configuration service block, and FIPS compliance. After the 30-day trial expires you are blocked from using enterprise connectors until you upgrade. Full per-feature config keys (including the logminer{} sub-block, the connector_list.yaml allow/deny structure, and --secrets URN schemes) are in references/enterprise-features.md.

Enterprise components (CDC inputs: postgres_cdc, mysql_cdc, mongodb_cdc, oracledb_cdc, microsoft_sql_server_cdc, aws_dynamodb_cdc, gcp_spanner_cdc, salesforce_cdc) require a Redpanda Enterprise license. Without one, these components fail at connection time with a license error. (tigerbeetle_cdc is the exception among CDC inputs: it is a certified community component — no license required — but it needs a CGO-enabled Connect build; it is absent from rpk connect and the standard Docker image, so a "component not found" error there is a build issue, not a license issue.)

Default license path: /etc/redpanda/redpanda.license (grounded in internal/license/service.go, constant defaultLicenseFilepath)

Environment variables:

• REDPANDA_LICENSE — inline license string
• REDPANDA_LICENSE_FILEPATH — path to license file

CLI flag:

rpk connect run --redpanda-license "$(cat redpanda.license)" ./pipeline.yaml
rpk connect dry-run --redpanda-license "$(cat redpanda.license)" ./pipeline.yaml

The license service logs on startup:

level=info msg="Successfully loaded Redpanda license" license_org=MyOrg license_type=enterprise expires_at=2026-01-01T00:00:00Z

If no valid Enterprise license is found, a 10-year open-source license is applied automatically. Attempting to use an Enterprise component under the open-source license produces an error at connection time (visible in dry-run).

Allow/deny list: /etc/redpanda/connector_list.yaml restricts which components an instance may run (allow: OR deny:, never both). If a component unexpectedly "does not exist," check this file. See references/enterprise-features.md.

Secrets management: the --secrets <urn> flag resolves ${SECRET} interpolations from a remote system at runtime. Schemes: env:, none:, aws:, gcp:, az:, redis:. URNs are tried in order; first hit wins.

Common Failure Modes

See references/failure-modes.md for a step-by-step triage checklist. Quick index:

Symptom	First move
Pipeline exits immediately	rpk connect lint then check stderr for the first ERROR line
/ready stuck at 503	Check input/output connection errors in DEBUG logs
Messages drop silently	Enable logger.level: DEBUG; look for failed to process or output errors
Enterprise component fails	Check license: REDPANDA_LICENSE env or --redpanda-license flag
TLS handshake failure	Verify tls.root_cas_file path and cert expiry
Auth / credential error	Use rpk connect dry-run --verbose to surface the exact connection error
High latency / backpressure	Check output_sent vs input_received in metrics; inspect buffer config
Memory growth	Enable http.debug_endpoints: true; capture heap profile

Reference Directory

• lint-and-validate.md: rpk connect lint and rpk connect dry-run flags, exit codes, what each catches, env-var interpolation handling, and CI integration patterns.
• logging-metrics-tracing.md: Full logger config (all levels and fields), all metrics backends (prometheus/statsd/json_api/logger) with complete YAML, OpenTelemetry tracer config, and the /ping//ready endpoint reference.
• failure-modes.md: Triage checklist for common Connect failure modes: connection/TLS/auth errors, checkpoint/cache failures, Enterprise license errors, backpressure and consumer lag, and memory pressure — each with the exact knob to turn.
• enterprise-features.md: Redpanda Connect enterprise features and their nested config keys (all require an Enterprise license): license loading order (--redpanda-license, REDPANDA_LICENSE, REDPANDA_LICENSE_FILEPATH, /etc/redpanda/redpanda.license); enterprise CDC inputs with full skeletons (postgres_cdc slot/heartbeat, mysql_cdc binlog/checkpoint_cache, mongodb_cdc change streams, oracledb_cdc logminer{} block); connector allow/deny lists (/etc/redpanda/connector_list.yaml); secrets management --secrets URN schemes; the redpanda: configuration service block; and FIPS compliance.