Installer avec Codex ou Claude Copiez ce prompt, collez-le dans Codex, Claude ou un autre assistant, puis laissez-le vérifier la page du skill et l'installer pour vous.
Une commande directe contourne le prompt de vérification. Examinez la source avant de l'exécuter.
Verified agent platforms: Claude / Qoder / OpenCode / Codex. Other platforms are not compatibility-tested.
⚠️ This skill performs production data writes and overwrites. Preview every write operation with --dry-run first, and execute only after human review. All output is for reference and must be verified by a human.
Safety Red Lines
Overwrites are irreversible: INSERT OVERWRITE clears all data in the target Paimon table; --force runs DROP TABLE then recreates. Before running, confirm the target table is empty, disposable, or already backed up.
Never commit credentials: OSS AK/SK, Spark password, and Metastore password in config.ini are highly sensitive. Never commit them to Git/SVN. Add config.ini to .gitignore and inject credentials via environment variables (see "Credential Security").
No real-prefix placeholders: Never write example AKs with real prefixes like LTAI.... Always use <YOUR_AK> / <YOUR_SK>.
: The default rclone concurrency of 64 may overwhelm the source HDFS or target OSS. On production clusters, evaluate with low concurrency first (e.g. ) before scaling up.
Large-cluster protection
--max-parallel 2 --transfers 16
DLS direct-read boundary: --direct-read only applies to OSS-HDFS (DLS) sources. Enabling it on plain HDFS results in unreadable data.
Writes require confirmation: Before running CREATE TABLE / DROP TABLE / INSERT OVERWRITE / rclone copy, the agent must show the impact and obtain explicit confirmation (see references/agent-rules.md).
Tooling Overview
This skill depends on no external MCP tools; it is implemented via a Python CLI plus external services:
Choose the execution path based on user input and environment:
Trigger condition
Recommended tool
Key flags
User already has an inspect output directory
main.py -e <explore_dir>
Recommended path
User only gives a database list (e.g. ads,dwd)
main.py -d ads,dwd
Needs [metastore_db] + hive CLI
User wants only a few specific tables
main.py -t db1.t1,db2.t2
Same as above
Source is OSS-HDFS / DLS (contains oss-dls.aliyuncs.com)
Add --direct-read
Skips rclone
Target database may not exist
Add --auto-create-db
Auto-creates DB (needs confirmation)
Target table exists but must be rebuilt
Add --force
DROP+CREATE (per-table confirmation)
User wants batched / resumable runs
--start-step N / --skip-steps a,b
Reuse the same --output-dir
Only DDL conversion, no execution
scripts/ddl_converter/cli.py
Connects to no external service
User already has an inspect incremental output
incremental_migrate.py -i <incr_dir>
Three-phase pipeline
Decision flow:
Check the input source first (explore dir / DB list / table list).
Check the source storage type (plain HDFS / DLS) to decide whether to enable --direct-read.
Check whether same-name tables/DBs exist on the target to decide --force / --auto-create-db.
Preview any write operation with --dry-run before execution.
Overview
Full Migration (main.py)
Step
Script
Key output
1. Generate Paimon internal-table DDL
step1_generate_paimon_ddl.py
paimon_ddl.sql + table_manifest.csv
2. Generate external-table DDL
step2_generate_ext_ddl.py
paimon_ext_ddl.sql + text_tables_insert.sql
3. Execute table-creation DDL
step3_execute_ddl.py
ddl_result.csv
4. rclone data sync
step4_rclone_sync.py
rclone_result.csv
5. INSERT OVERWRITE
step5_insert_overwrite.py
insert_result.csv
The orchestrator main.py chains Step 1-5, supporting --start-step for resumable runs and --skip-steps to skip specific steps.
Incremental Migration (incremental_migrate.py)
Phase
Content
Key output
Phase 1
Execute DDL (CREATE TABLE)
incr_ddl_result.csv
Phase 2
rclone data sync
incr_rclone_result.csv
Phase 3
INSERT OVERWRITE data load
incr_insert_result.csv
The standalone incremental_migrate.py executes commands generated by the migration-lhm-inspect-hive-metastore incremental exploration, supporting parallel and background execution.
Quick Start
Edit config.ini with the real connection info for your environment (Metastore DB, HDFS, OSS, Spark Thrift). Full field reference: references/configuration.md.
Run one of:
# Run a full migration using migration-lhm-inspect-hive-metastore output
python main.py -e /path/to/hive_explore_all_dbs_YYYYMMDD/ -c config.ini
# Or migrate specific databases
python main.py -d ads,dwd,dws -c config.ini
# dry-run preview (does not execute)
python main.py -e /path/to/explore/ -c config.ini --dry-run
# direct-read mode (OSS-HDFS/DLS; external table points at source path, skips rclone)
python main.py -e /path/to/explore/ -c config.ini --direct-read
🔐 All config examples use <...> placeholders. In production, inject credentials via environment variables and add config.ini to .gitignore.
The explore directory must contain summary_report.csv and a ddl_files/ subdirectory. Use --filter-db and --filter-tables to further filter the explore results.
Source B: specify databases or table names
python main.py -d ads,dwd -c config.ini # by database
python main.py -t ads.ads_xxx,dwd.dwd_yyy -c config.ini # by table
This mode requires the [metastore_db] config and a usable hive CLI on the ECS host.
Step Details
Step 1: Generate Paimon internal-table DDL
Converts Hive DDL into Paimon internal-table CREATE statements.
Conversion rules:
Remove Hive storage info: ROW FORMAT / STORED AS / LOCATION / TBLPROPERTIES.
Add USING paimon.
Merge partition columns into the column list (required by Paimon).
PARTITIONED BY keeps only column names, not types.
Generates and runs INSERT OVERWRITE to load external-table data into the Paimon internal tables. TextFile tables use the special INSERT statement generated in Step 2.
Paimon format-table cannot read multi-column TextFile data directly. Solution:
External table: create a single-column raw_line string external table (suffix _oss) with file.format = 'text'.
INSERT: use split(raw_line, '\u0001') to split fields, CAST to convert types, and CASE WHEN ... = '\\N' THEN NULL for nulls.
This is handled automatically in Step 2; no manual intervention needed.
Direct-Read Mode (OSS-HDFS/DLS)
When source data is on OSS-HDFS (DLS), rclone cannot access the DLS data layer via the S3 API (DLS and plain OSS are different storage layers). Use --direct-read mode:
The Step 2 external-table DDL uses the source DLS path directly (e.g. oss://bucket.cn-hangzhou.oss-dls.aliyuncs.com/...).
Step 4 (rclone) is skipped automatically.
Spark EMR has a built-in DLS driver and can read DLS paths directly.
# Python dependencies
pip install pyhive thrift thrift_sasl
# Only for -d/-t input modes (connect to Metastore DB)
pip install PyMySQL # MySQL Metastore
pip install psycopg2-binary # PostgreSQL Metastore# rclone (data sync tool) — the script auto-detects and tries to install it# CentOS/RHEL: yum install -y epel-release && yum install -y rclone# Debian/Ubuntu: apt-get install -y rclone# Generic: curl https://rclone.org/install.sh | bash
Troubleshooting
See references/troubleshooting.md, covering 14 common error classes (preflight, Spark connection, DDL execution, rclone sync, TextFile INSERT, DLS access, EMR Gateway 401, AK/SK leaks, etc.) and how to diagnose them.
Agent Execution Rules
See references/agent-rules.md, which covers direct-read auto-detection, the rclone parameter-confirmation flow (with AK/SK masking), and the write-operation confirmation mechanism.
Incremental Migration (incremental_migrate.py)
Prerequisites
First generate an output directory via the incremental exploration of the migration-lhm-inspect-hive-metastore skill, containing:
sync_commands.sh — rclone data-sync commands.
paimon_sync.sql — Paimon table-creation and data-load SQL.
Disclaimer: > This output is based on automated migration-script results. Whether the source Hive tables can be decommissioned must be decided after human review and business-side validation.
Output example:
[Migration complete] Mode: standard | Scope: 3 DBs / 27 tables
✅ Success: 25 ❌ Failed: 2 ⏭ Skipped: 0
Report: output/20260514_103045/
Top failures: ① TextFile INSERT error (1) ② OSS auth failure (1)
Suggestion: python main.py ... --start-step 5 --filter-tables ads.t1,dwd.t2
⚠️ Please review row counts and sample data manually before decommissioning source Hive tables.
