tidb2snowflake-local-e2e

name

tidb2snowflake-local-e2e

description

Run and debug the tidb2snowflake local end-to-end path that avoids TiDB Cloud: start a local TiDB cluster with TiUP playground, seed TiDB data, dump snapshot CSV to S3 with Dumpling, stream incremental CSV to S3 with TiCDC, then run tidb2snowflake to load and merge into Snowflake. Use when asked to test tidb2snowflake locally, reproduce the dumping + CDC to S3 flow, verify Snowflake loading, or explain/run the local_tiup_s3_snowflake_e2e.sh workflow.

tidb2snowflake Local E2E

Core Rule

Do not use TiDB Cloud for this workflow. The local path is:

TiUP playground TiDB -> Dumpling snapshot CSV -> S3 snapshot/
TiUP playground TiCDC -> S3 increment/
S3 snapshot/ + increment/ -> tidb2snowflake -> Snowflake

tidb2snowflake skips TiDB Cloud OpenAPI when the S3 root already contains objects under both snapshot/ and increment/.

Quick Start

Run from the nested repo:

cd /Users/jiangjianyuan/work/git/pingcap/tidb2snowflake
./scripts/local_tiup_s3_snowflake_e2e.sh

The script reads tidb2snowflake/.env for S3 and Snowflake credentials. It creates a unique S3 prefix and Snowflake schema per run, builds bin/tidb2snowflake, starts tiup playground v8.5.6 --ticdc 1, seeds one table, dumps snapshot data, creates a TiCDC cloud-storage changefeed, applies DMLs, starts the loader, and polls Snowflake for the final state.

Useful overrides:

RUN_ID=manual1 \
TIDB_VERSION=v8.5.6 \
PORT_OFFSET=22000 \
SNOWFLAKE_DATABASE=TIDB2SNOWFLAKE_E2E \
SNOWFLAKE_SCHEMA=LOCAL_MANUAL1 \
./scripts/local_tiup_s3_snowflake_e2e.sh

Use SNOWFLAKE_ACCOUNT_ID, SNOWFLAKE_USER, SNOWFLAKE_PASS, , , , , and to override .