clickzetta-cdc-sync-pipeline

Stars7

Forks3

UpdatedJune 2, 2026 at 08:10

Create and manage ClickZetta Lakehouse multi-table real-time sync (CDC) tasks, syncing MySQL/PostgreSQL databases to Lakehouse. Three sync modes: full database mirror, multi-table mirror, sharded table merge. Binlog/WALs-based with full load + incremental sync. Triggered when the user says "multi-table real-time sync", "full database sync", "database mirror", "CDC full database", "multi-table CDC", "sharded table merge", "MySQL full database sync", "PostgreSQL full database sync", "database migration", "sync operations", "sync SOP", "Binlog position expired", "full re-sync", "add sync table". Covers source DB preparation, sync mode selection, task deployment, operations SOP, monitoring/alerting, and troubleshooting — all ClickZetta Studio specific logic. Keywords: CDC, real-time sync, MySQL, PostgreSQL, change data capture, mirror, merge, multi-table

Installation

Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.

Source

clickzetta

clickzetta/clickzetta-skills

View GitHub Repository View Creator Repositories

Download

Related occupationsSOC

Based on SOC occupation classification

Software DevelopersComputer and Mathematical Occupations·SOC 15-1252

File Explorer

3 files

SKILL.md

readonly

More from this repository

same repository

clickzetta-analytics-agent

clickzetta/clickzetta-skills

Complete guide for managing ClickZetta Analytics Agent domains via cz-cli: domain CRUD, table registration, column semantics (types/descriptions/aliases), metrics (simple & compound), Answer Builders, Knowledge Bases, and QA testing. Wizard-driven workflows with intent classification, routing tables, and troubleshooting references. Trigger when the user says: "analytics agent", "分析域", "语义层", "指标", "answer builder", "知识库", "多别名", "column semantics", "domain prompt", "metric create", "AB create", "session run", "domain table add", "semantics set", "knowledge space". Keywords: analytics agent, domain, semantics, metric, answer builder, knowledge base, alias, cz-cli, data analysis, AI agent, semantic layer

2026-07-317

clickzetta-semantic-view

clickzetta/clickzetta-skills

Create, query, and manage ClickZetta Lakehouse Semantic Views — schema-level logical models that encapsulate multi-table JOINs and aggregations into a business-friendly layer of logical tables, dimensions, metrics, and facts. Query with the semantic_view() function without writing JOINs or GROUP BY manually. Triggered when user says "create semantic view", "semantic view", "semantic layer", "define metrics", "define dimensions", "unified metric definitions", "business semantic model", "semantic_view()", "CREATE OR REPLACE SEMANTIC VIEW", "FACTS", "PRIVATE metric", "conditional metric", "window metric", "SHOW SEMANTIC VIEWS", "GRANT SELECT ON SEMANTIC VIEW". Keywords: semantic view, dimension, metric, fact, logical model, unified metrics, semantic layer, grain, chasm trap, FILTER metric

2026-07-317

clickzetta-data-science

clickzetta/clickzetta-skills

End-to-end data science workflow guide for ClickZetta Lakehouse, covering environment setup, data discovery, feature engineering (SQL + ZettaPark), and model inference deployment. Details: Python 3.10+/Jupyter/ZettaPark setup, project structure, data quality assessment, and inference (BITMAP profiling, UDF batch inference, vector search). Trigger when the user wants to do data science, ML, or analytical work using ClickZetta Lakehouse — connecting Jupyter to Lakehouse, doing EDA, building features, running ML inference, user profiling, audience segmentation, or batch scoring. Keywords: data science, ML, ZettaPark, Jupyter, feature engineering, EDA, profiling, inference

2026-07-317

clickzetta-zettapark

clickzetta/clickzetta-skills

Use the ZettaPark Python library to work with ClickZetta Lakehouse data through a pandas-like DataFrame API that translates Python transformations into distributed SQL — no manual SQL needed. Covers Session creation, DataFrame construction and transformation (filter/select/join/groupBy), result collection (collect/to_pandas/show), writing tables (save_as_table), file operations (PUT/GET), and executing SQL. Trigger when the user says: "ZettaPark", "zettapark", "DataFrame API", "Python Lakehouse", "save_as_table", "session.table", "session.sql", "collect()", "to_pandas", "Python data engineering", "Python write to Lakehouse", "Python read from Lakehouse", "clickzetta_zettapark_python", "Python ETL", "Python ML on Lakehouse", "feature engineering Python", "Python DataFrame Lakehouse". Keywords: ZettaPark, DataFrame, pandas-like, Python, SQL translation, distributed compute

2026-07-317

singsight-install

clickzetta/clickzetta-skills

Guide users to connect their AI coding agents (Claude Code, Hermes Agent, OpenClaw, Opencode) to the Singsight AI Agent observability platform. Automatically reports every LLM call, tool execution, token usage, and session via OpenTelemetry. Triggered when the user says "connect Singsight", "install Singsight", "configure Singsight", "Agent observability", "OTel integration", "report token usage", "monitor AI Agent", "singsight install". Keywords: singsight, install, observability, opentelemetry, otel, agent telemetry, claude code, hermes, openclaw, opencode

2026-07-027

clickzetta-dbt-modeling

clickzetta/clickzetta-skills

dbt-clickzetta data modeling wizard: discovers Lakehouse data sources, infers modeling strategies, generates sources.yml and model files, and runs the full test suite. Presents choices rather than blank forms — explores data first, then proposes a justified plan for user confirmation. Trigger when the user wants to turn raw Lakehouse data into usable analytical tables — mentions dbt (dbt modeling, dbt model, sources.yml, incremental, dbt run, dbt test), wants ETL transformation, dimension/fact tables, or asks "how do I turn these tables into a queryable dataset". Keywords: dbt, dbt modeling, sources.yml, incremental, dbt run, dbt test, dimension table, fact table, ETL

2026-07-017

name

clickzetta-cdc-sync-pipeline

description

Multi-table Real-time Sync (CDC) Pipeline Workflow

See references/troubleshooting.md for troubleshooting, cz-cli alternative path, and delivery checklist.

Wizard: Collect Required Information

Ask the user three questions: (1) Source database type — MySQL (Binlog-based) or PostgreSQL (WALs-based, requires v14+)? (2) Sync mode — full database mirror, multi-table mirror, or sharded table merge? (3) Is the source database already prepared (MySQL: Binlog enabled + REPLICATION permission; PG: wal_level=logical)?

Also confirm the target schema (e.g., ods).

If the user has already provided sufficient information, proceed directly without showing the menu.

Comparison with Other Sync Methods

Dimension	Multi-table Real-time Sync (This Skill)	Single-table Real-time Sync	Batch Sync
Task Type ID	`281`	`28`	`10` / `291`
Sync Granularity	Full database / multi-table / sharded merge	Single table/topic	Single/multi-table
Run Mode	Continuously running (streaming CDC)	Continuously running	Scheduled (batch)
Applicable Skill	`clickzetta-cdc-sync-pipeline`	`clickzetta-realtime-sync-pipeline`	`clickzetta-batch-sync-pipeline`

Prerequisites

Source database prepared with CDC permissions (see Source Database Preparation below)
Source data source configured in Studio UI (not via SQL CREATE STORAGE CONNECTION — that only supports object storage and Kafka)
Sync VCluster available (task_type=281 requires a Sync VCluster)
cz-cli or MCP tools available

Environment Detection

cz-cli --version   # if exists → use cz-cli path (see references/troubleshooting.md)

If cz-cli not found, check if MCP list_data_sources tool is available. If neither is available, prompt the user to install one.

Source Database Preparation

MySQL

Parameter	Required Value	Query
`log_bin`	ON	`SHOW GLOBAL VARIABLES LIKE 'log_bin'`
`binlog_format`	ROW	`SHOW GLOBAL VARIABLES LIKE 'binlog_format'`
`binlog_row_image`	FULL	`SHOW GLOBAL VARIABLES LIKE 'binlog_row_image'`
`binlog_expire_logs_seconds`	≥86400	—

Permissions: SELECT on information_schema + target tables, REPLICATION SLAVE, REPLICATION CLIENT.

PostgreSQL

Parameter	Required Value
`wal_level`	logical
`max_replication_slots`	≥10
`max_wal_senders`	≥10

Permissions: SELECT on information_schema, REPLICATION, CREATE (for publication).

PostgreSQL: each task needs its own replication slot — never reuse slots across tasks.

Workflow

Step 1: Confirm Sync VCluster

LH_show_object_list(object_type='VCLUSTERS') → filter for vcluster_type containing SYNC

Step 2: Find Source Data Source

list_data_sources → filter by type (MySQL: ds_type=5, PostgreSQL: ds_type=7)
Record datasource_id and datasource_type.

Step 3: Explore Source Data Structure

list_namespaces → view source database list
list_metadata_objects → view tables under a database
Confirm sync scope (full database / specific tables / sharded tables).

Step 4: Select Sync Mode

Mode	pipeline_type	Use Case
Full database mirror	3	Sync all tables, auto-adapts to new tables
Multi-table mirror	1	Sync selected tables, supports schema change detection
Sharded table merge	2	Merge sharded tables into a single target table

Step 5: Create Task

create_task(task_type=281, task_name="cdc_sync_<database>", data_folder_id=<folder_id>)
Record returned task_id.

Step 6: Configure Sync Content

save_cdc_realtime_task(
  data_file_id=<task_id>,
  pipeline_type=<mode from Step 4>,
  source_datasource_list=[{"datasourceId": <id>, "datasourceType": <type>}],
  sync_object_list=<tables or database>,
  target_datasource={"datasourceId": <lakehouse_id>, "datasourceType": 1},
  sync_mode=1,   # 1=full load + incremental (recommended), 2=incremental only
  save_mode=2    # 2=append (recommended for new tasks)
)

Step 7: Submit and Deploy

publish_task(task_id=<id>, task_version=<version>)

CDC tasks do not start automatically after submission — start manually in Studio UI.

Step 8: Start the Task

Start Method	Use Case
Stateless start	First start (full load → incremental)
Resume from last saved state	Restart after stop
Custom start position	Data re-sync (specify binlog file/time or LSN)

Step 9: Monitor

After starting, the task goes through: Initialization → Full Load → Incremental Sync.

Key metrics: data read/written, average read/write rate, failover count, per-table latency.

Per-table operations: priority execution, re-sync, backfill sync, view exceptions.

Three Sync Modes in Detail

Full Database Mirror

Database-level granularity — select database only, not individual tables
Auto-adapts to new tables added to the database

Multi-table Mirror

Table-level selection, supports automatic column add/drop detection
PostgreSQL requires replication slot configuration (decoderbufs or pgoutput plugin)

Sharded Table Merge

Merges sharded tables into a single target table via "virtual tables"
Two configuration methods: rule-based (regex) or file-based (upload config file)
Enable extended fields to record source info (__source_table__) and avoid primary key conflicts

Operations SOP

Add Sync Tables

Edit task, add tables, save
Submit for deployment
Stop then restart — new tables sync automatically (full load if configured)
Does not affect existing tables' sync progress

Single-table Data Repair

Operation	Write Method
Re-sync	Sync to temp table → insert overwrite to target
Backfill sync	Sync to temp table → delete related data → merge into target

Monitoring and Alerting

Configure these 5 alert rules:

Alert Type	Monitored Item
Task Failover	Multi-table real-time sync job failover
Task Stopped	Multi-table real-time sync task run failure
Single-table Exception	Target table change failure (Schema Evolution, field size exceeded)
End-to-end Latency	Time from source to target
Read Position Lag	Gap between read position and source latest position

IM webhook: configure Feishu/WeCom group bot → add webhook URL in product → enable in notification policy.

Known Limitations

Cannot create MySQL/PostgreSQL Connection via SQL (CREATE STORAGE CONNECTION TYPE MYSQL will error)
Schema Evolution does not support column type changes
Only tables with primary key (PK) fields are supported
Do not manually create/modify/delete target tables (system auto-manages structure)
MySQL unsupported: year column type
PostgreSQL unsupported: varbit, bytea, TIMETZ, interval, NAME, NUMERIC/decimal (precision mismatch)