| name | connect-cdc-tigerbeetle |
| description | Guides setup and operation of the tigerbeetle_cdc Redpanda Connect input for streaming change data capture from a TigerBeetle financial transactions database into Redpanda or Kafka. Covers the cgo-enabled binary requirement, connecting to a TigerBeetle cluster (cluster_id, replica addresses), the progress_cache checkpointing model, resuming from a timestamp (timestamp_initial), the JSON change-event shape (transfer + debit/credit account snapshots), and per-event-type or per-ledger routing. Use when: capturing TigerBeetle ledger events (transfers, account updates) into Redpanda or Kafka, configuring tigerbeetle_cdc, choosing a persistent progress cache, filtering settled vs pending two-phase transfers with the event_type metadata, routing by ledger, archiving transfer events to S3, troubleshooting "component not available" (cgo builds) or duplicate events after restart, or asking whether tigerbeetle_cdc needs an Enterprise license (it does not — it is a certified community connector, unlike the other CDC inputs). Also covers the Redpanda Enterprise features the CDC destination topic and cluster can use: Iceberg Topics (redpanda.iceberg.mode/delete/partition.spec/target.lag.ms/ invalid.record.action), Tiered Storage (redpanda.remote.read/write) for long-term ledger-event retention, Remote Read Replicas, Shadow Linking (rpk shadow cross-cluster DR), server-side Schema ID Validation, RBAC, Audit Logging, and OIDC/Kerberos auth on the landed topic. |
Redpanda Connect CDC: TigerBeetle
The tigerbeetle_cdc input streams change events from a TigerBeetle cluster — the purpose-built financial transactions database — into Redpanda or any Kafka-compatible broker. Every event is a JSON snapshot of a transfer plus its debit and credit accounts at the time of the event. Progress is checkpointed as the last acknowledged event timestamp in a Connect cache resource, so the pipeline resumes where it left off after a restart. Available in the docs from Redpanda Connect v4.65.0, currently marked beta (API subject to change). Unlike every other CDC input, this is a certified (community-tier) connector, not an enterprise one: internal/plugins/info.csv marks it certified, the source carries an Apache-2.0 license header, and there is no runtime Enterprise license check.
Two constraints set this connector apart from the other CDC inputs:
- cgo-only builds. The component is compiled only into cgo-enabled builds of Redpanda Connect. It is not available in the
rpk connect CLI or the standard Docker image — download the prebuilt cgo-enabled binary (Linux AMD64 only) or build from source with cgo enabled.
- No snapshot mode. There is no separate initial-snapshot phase: by default the connector streams all CDC events available in the TigerBeetle cluster from the beginning, or from
timestamp_initial if set. Each event already embeds the full current state of the transfer and both accounts.
Quickstart
1. Get a cgo-enabled Redpanda Connect binary
wget https://github.com/redpanda-data/redpanda-connect/releases/download/v<VERSION>/redpanda-connect-cgo_<VERSION>_linux_amd64.tar.gz
tar -xzf redpanda-connect-cgo_<VERSION>_linux_amd64.tar.gz
sudo mv redpanda-connect /usr/local/bin/
redpanda-connect --version
Requires a TigerBeetle cluster version 0.16.57 or later, and the Connect TigerBeetle client version must not be newer than the cluster version.
2. Create the pipeline config
The input requires a cache resource (progress_cache) to store the last acknowledged event timestamp. Use a persistent cache (Redis, aws_dynamodb, sql) in production — with an in-memory cache, every restart re-streams from the beginning.
input:
tigerbeetle_cdc:
cluster_id: "1"
addresses:
- "192.168.1.10:3000"
- "192.168.1.11:3000"
- "192.168.1.12:3000"
progress_cache: redis_cache
output:
redpanda:
seed_brokers: ["localhost:9092"]
topic: 'transfers.${! meta("ledger") }'
key: ${! json("transfer.id") }
cache_resources:
- label: redis_cache
redis:
url: redis://localhost:6379
3. Run it
redpanda-connect run pipeline.yaml
4. Inspect messages
Each message body is a JSON object with the event timestamp, event type, ledger, and full snapshots of the transfer and both accounts. 128-bit and 64-bit numeric fields are serialized as decimal strings; 32-bit and 16-bit fields are JSON numbers:
{
"timestamp": "1745328372758695656",
"type": "single_phase",
"ledger": 2,
"transfer": {
"id": "9082709",
"amount": "3794",
"pending_id": "0",
"user_data_128": "79248595801719937611592367840129079151",
"user_data_64": "13615171707598273871",
"user_data_32": 3229992513,
"timeout": 0,
"code": 20295,
"flags": 0,
"timestamp": "1745328372758695656"
},
"debit_account": {
"id": "3750",
"debits_pending": "0",
"debits_posted": "8463768",
"credits_pending": "0",
"credits_posted": "8861179",
"user_data_128": "118966247877720884212341541320399553321",
"user_data_64": "526432537153007844",
"user_data_32": 4157247332,
"code": 1,
"flags": 0,
"timestamp": "1745328270103398016"
},
"credit_account": { "...": "same shape as debit_account" }
}
Every message carries these metadata fields (all values are strings):
| Metadata key | Value |
|---|
event_type | single_phase, two_phase_pending, two_phase_posted, two_phase_voided, or two_phase_expired |
ledger | The ledger code |
transfer_code | The transfer code |
debit_account_code | The debit account code |
credit_account_code | The credit account code |
timestamp | Unique event timestamp, nanosecond resolution |
timestamp_ms | Event timestamp, millisecond resolution |
Progress and Resume Semantics
- Checkpoint: after each batch is acknowledged, the connector writes the last event timestamp to
progress_cache under the key timestamp_last_<cluster_id>. On restart it resumes from the next event.
timestamp_initial: a TigerBeetle nanosecond timestamp to start from (inclusive). Ignored if the cache already holds a more recent acknowledged timestamp. Unset = stream everything available in the cluster.- Ordering: events are delivered strictly in order — the connector holds the next batch until the current one is acknowledged.
- Delivery: at-least-once. During crash recovery, unacknowledged messages may be replayed; consumers must be idempotent. The nanosecond
timestamp metadata is unique per event and suitable for deduplication.
Event Types and Filtering
TigerBeetle transfers are either single-phase (settled immediately) or two-phase (pending, then posted/voided/expired). Keep only settled transfers:
pipeline:
processors:
- mapping: |
root = if meta("event_type") != "single_phase" && meta("event_type") != "two_phase_posted" {
deleted()
}
Per-Event-Type Routing
output:
switch:
cases:
- check: meta("event_type") == "single_phase"
output:
redpanda:
seed_brokers: ["localhost:9092"]
topic: tigerbeetle.single_phase
- check: meta("event_type") == "two_phase_posted"
output:
redpanda:
seed_brokers: ["localhost:9092"]
topic: tigerbeetle.two_phase
- output:
drop: {}
Operational Notes
- Component not available: if
tigerbeetle_cdc is not recognized, you are running a non-cgo build (rpk connect or the standard Docker image). Switch to the cgo-enabled binary.
- Persistent cache: the progress cache is the only durable state. Losing it re-streams all available events (safe but duplicative under at-least-once).
- Throughput/pacing:
event_count_max (default 2730) caps events per request; idle_interval_ms (default 1000) is the wait when a poll returns nothing; an optional rate_limit resource throttles requests; timeout_seconds (default 15) bounds each query.
- Version: verify field names and defaults against the generated reference (
modules/components/partials/fields/inputs/tigerbeetle_cdc.adoc in rp-connect-docs) or redpanda-connect create tigerbeetle_cdc on a cgo-enabled binary — not rpk connect create, which lacks the component.
- Support tier / license:
certified in internal/plugins/info.csv — the only CDC input that is not enterprise there. Apache-2.0 source, no enterprise license check. Status is beta — the API may change.
- Redpanda Cloud:
info.csv marks the component cloud: n ("not yet certified for cloud") — it cannot run as a Redpanda Cloud managed pipeline. Run the self-hosted cgo binary and write to your Cloud cluster over TLS/SASL.
Enterprise Features on the Destination Topic
The tigerbeetle_cdc input is a certified community connector — no Enterprise license is needed for the input itself — but the Redpanda topic and cluster the ledger events land in can use Redpanda Enterprise differentiators (each requires a valid Redpanda Enterprise license on the destination cluster):
- Iceberg Topics — land transfer events directly into an Apache Iceberg table for analytics. Per-topic:
redpanda.iceberg.mode (key_value | value_schema_id_prefix | value_schema_latest | disabled), redpanda.iceberg.delete, redpanda.iceberg.partition.spec, redpanda.iceberg.target.lag.ms, redpanda.iceberg.invalid.record.action (drop | dlq_table); cluster: iceberg_enabled.
- Tiered Storage —
redpanda.remote.write + redpanda.remote.read (cluster cloud_storage_enabled) give the ledger-event topic effectively unlimited retention for audit and compliance.
- Remote Read Replicas —
redpanda.remote.readreplica for read-only, object-storage-served copies of the CDC topic in a remote cluster.
- Shadow Linking / Shadowing — offset-preserving cross-cluster DR for the CDC topic, managed with the
rpk shadow family (create/list/describe/status/update/failover/delete/config-generate).
- Server-side Schema ID Validation —
redpanda.value.schema.id.validation + redpanda.value.subject.name.strategy (cluster enable_schema_id_validation = none/redpanda/compat).
- RBAC, Audit Logging, OIDC/Kerberos, FIPS — secure the CDC topic and pipeline; audit logging is a natural fit for financial event streams.
- Redpanda Connect Enterprise (separate Connect license) — secrets management, configuration service, allow/deny lists.
See Enterprise Redpanda Features for grounded nested config keys, defaults, license-expiration behavior, and rpk commands.
Reference Directory
- Config Reference: Every
tigerbeetle_cdc config field with type, default, and validation rule, grounded in source.
- TigerBeetle Setup: cgo-enabled binary acquisition, TigerBeetle version requirements, cluster connection parameters, and progress-cache selection.
- Pipeline and Output: Full runnable pipelines, message/metadata shape, event-type semantics, routing patterns (Redpanda, S3), resume-from-timestamp, and restart behavior.
- Enterprise Redpanda Features: Iceberg Topics, Tiered Storage, Remote Read Replicas, Shadow Linking (cross-cluster DR), server-side Schema ID Validation, RBAC/Audit Logging/OIDC/Kerberos/FIPS, and Redpanda Connect Enterprise — nested config keys, defaults, and which license gates each.
