tidb2snowflake-local-e2e

name

tidb2snowflake-local-e2e

description

Run and debug the tidb2snowflake local end-to-end path that avoids TiDB Cloud: start a local TiDB cluster with TiUP playground, seed TiDB data, dump snapshot CSV to S3 with Dumpling, stream incremental CSV to S3 with TiCDC, then run tidb2snowflake to load and merge into Snowflake. Use when asked to test tidb2snowflake locally, reproduce the dumping + CDC to S3 flow, verify Snowflake loading, or explain/run the local_tiup_s3_snowflake_e2e.sh workflow.

tidb2snowflake Local E2E

Core Rule

Do not use TiDB Cloud for this workflow. The local path is:

TiUP playground TiDB -> Dumpling snapshot CSV -> S3 snapshot/
TiUP playground TiCDC -> S3 increment/
S3 snapshot/ + increment/ -> tidb2snowflake -> Snowflake

tidb2snowflake skips TiDB Cloud OpenAPI when the S3 root already contains objects under both snapshot/ and increment/.

Quick Start

Run from the nested repo:

cd /Users/jiangjianyuan/work/git/pingcap/tidb2snowflake
./scripts/local_tiup_s3_snowflake_e2e.sh

The script reads tidb2snowflake/.env for S3 and Snowflake credentials. It creates a unique S3 prefix and Snowflake schema per run, builds bin/tidb2snowflake, starts tiup playground v8.5.6 --ticdc 1, seeds one table, dumps snapshot data, creates a TiCDC cloud-storage changefeed, applies DMLs, starts the loader, and polls Snowflake for the final state.

Useful overrides:

RUN_ID=manual1 \
TIDB_VERSION=v8.5.6 \
PORT_OFFSET=22000 \
SNOWFLAKE_DATABASE=TIDB2SNOWFLAKE_E2E \
SNOWFLAKE_SCHEMA=LOCAL_MANUAL1 \
./scripts/local_tiup_s3_snowflake_e2e.sh

Use SNOWFLAKE_ACCOUNT_ID, SNOWFLAKE_USER, SNOWFLAKE_PASS, SNOWFLAKE_WAREHOUSE, STORAGE_URI, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN to override .env.

Expected Evidence

A successful S3 preparation has objects like:

s3://.../local-<run>/snapshot/metadata
s3://.../local-<run>/snapshot/<db>.<table>.000000000.csv
s3://.../local-<run>/increment/metadata
s3://.../local-<run>/increment/<db>/<table>/<table-version>/meta/schema_<version>_<hash>.json
s3://.../local-<run>/increment/<db>/<table>/<table-version>/<yyyy-mm-dd>/CDC00000000000000000001.csv

The loader log should include:

snapshot data already exists in storage, skipping export creation
increment data already exists in storage, skipping changefeed creation

Those two lines prove the run is not using TiDB Cloud OpenAPI.

Debugging

Logs are written to tidb2snowflake/.local-e2e/<run-id>/:

playground.log
dumpling.log
changefeed.toml
changefeed-create.log
tidb2snowflake.log
snowflake_probe.go

For TiCDC internals, inspect:

~/.tiup/data/tidb2snowflake-<run-id>/cdc-0/ticdc.log

Common fixes:

• If changefeed-id is invalid, use only letters, digits, and hyphens.
• If TiCDC cannot access S3, ensure AWS env vars are exported before starting playground; the CDC server, not only cdc cli, needs credentials.
• If aws s3 ls finds no increment files, check changefeed-create.log, then query the changefeed with tiup cdc cli changefeed query --server http://127.0.0.1:<cdc-port> -c <id>.
• If Go fails with invalid reference to runtime.buildVersion, build/run Go helpers with -ldflags=-checklinkname=0.
• If Snowflake returns 390100 Incorrect username or password, the S3 and local TiDB/TiCDC path can still be valid; replace or override Snowflake credentials before retesting the final COPY/MERGE phase.

Manual Commands

Use the script by default. If debugging manually, mirror its important details:

tiup playground v8.5.6 --tag tidb2snowflake-<run> --db 1 --pd 1 --kv 1 --ticdc 1 --tiflash 0 --without-monitor --port-offset 21000

Dump snapshot:

tiup dumpling -h 127.0.0.1 -P 25000 -u root \
  --filetype csv --no-header --no-schemas \
  --csv-output-dialect snowflake --escape-backslash=false \
  --tables-list tidb2snowflake_local.t_<run> \
  --output "s3://<bucket>/<prefix>/snapshot" \
  --s3.region us-west-2 \
  --output-filename-template '{{.DB}}.{{.Table}}.{{.Index}}'

Create changefeed with protocol=csv, date-separator=day, include-commit-ts=true, and binary-encoding-method='hex'. Apply DML only after changefeed creation, then wait for increment files before starting tidb2snowflake.