メニュー
Apache SeaTunnel - A multimodal, high-performance, distributed data integration tool for massive data synchronization across 100+ connectors
デフォルトでは、最初にソースを確認する Prompt が選択されています。直接コマンドに切り替えるか、ローカルコピーをダウンロードすることもできます。
インストールを決める前に、SKILL.md と SkillsMP に表示されている付属ファイルをお読みください。
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
直接コマンドでは確認用 Prompt が省略されます。実行前にソースを確認してください。
npx skills add https://github.com/apache/seatunnel-tools --skill seatunnel-skillコマンドは1行のまま表示されます。コピー前に横へスクロールして全体を確認してください。
ローカルで確認しますか？SkillsMP が現在取得できるファイルをダウンロードできます。
SOC 職業分類に基づく
SKILL.md を表示中
| name | seatunnel-skill |
| description | Apache SeaTunnel - A multimodal, high-performance, distributed data integration tool for massive data synchronization across 100+ connectors |
| author | auto-generated by repo2skill |
| platform | github |
| source | https://github.com/apache/seatunnel |
| website_source | https://github.com/apache/seatunnel-website |
| tags | ["data-integration","data-pipeline","etl","elt","real-time-streaming","batch-processing","cdc","distributed-computing","apache","java","docusaurus","documentation"] |
| version | 2.3.13 |
| generated | "2026-02-26T00:00:00.000Z" |
| license | Apache 2.0 |
| repository | apache/seatunnel |
| website_repository | apache/seatunnel-website |
Apache SeaTunnel is a multimodal, high-performance, distributed data integration tool capable of synchronizing vast amounts of data daily. It connects hundreds of evolving data sources with support for real-time, CDC (Change Data Capture), and full database synchronization.
git clone https://github.com/apache/seatunnel.git
cd seatunnel
# Build entire project
mvn clean install -DskipTests
# Build specific module
mvn clean install -pl seatunnel-core -DskipTests
Visit the official download page and select your version:
VERSION=2.3.12
wget https://archive.apache.org/dist/seatunnel/${VERSION}/apache-seatunnel-${VERSION}-bin.tar.gz
tar -xzf apache-seatunnel-${VERSION}-bin.tar.gz
cd apache-seatunnel-${VERSION}
# Set JAVA_HOME
export JAVA_HOME=/path/to/java
# Add to PATH
export PATH=$PATH:/path/to/seatunnel/bin
seatunnel --version
Create a simple data integration job:
config/hello_world.conf
env {
job.mode = "BATCH"
job.name = "Hello World"
}
source {
FakeSource {
row.num = 100
schema = {
fields {
id = "bigint"
name = "string"
age = "int"
}
}
}
}
sink {
Console {
format = "json"
}
}
Run the job:
seatunnel.sh -c config/hello_world.conf -e spark
# or with flink
seatunnel.sh -c config/hello_world.conf -e flink
SeaTunnel is designed for:
Multimodal Support
Multiple Synchronization Methods
100+ Pre-built Connectors
Multi-Engine Support
# 1. Download binary
VERSION=2.3.12
wget https://archive.apache.org/dist/seatunnel/${VERSION}/apache-seatunnel-${VERSION}-bin.tar.gz
# 2. Extract
tar -xzf apache-seatunnel-${VERSION}-bin.tar.gz
cd apache-seatunnel-${VERSION}
# 3. Verify
./bin/seatunnel.sh --version
# 1. Clone repository
git clone https://github.com/apache/seatunnel.git
cd seatunnel
# 2. Build with Maven
mvn clean install -DskipTests
# 3. Navigate to distribution
cd seatunnel-dist/target/apache-seatunnel-*-bin/apache-seatunnel-*
# 4. Verify
./bin/seatunnel.sh --version
# Pull official Docker image
docker pull apache/seatunnel:latest
# Run container
docker run -it apache/seatunnel:latest /bin/bash
# Or run a job directly
docker run -v /path/to/config:/config \
apache/seatunnel:latest \
seatunnel.sh -c /config/job.conf -e spark
# Bash/Zsh
export JAVA_HOME=/path/to/java
export PATH=$JAVA_HOME/bin:$PATH
# Verify
java -version
# Set Spark Home (if using Spark engine)
export SPARK_HOME=/path/to/spark
# Verify Spark installation
$SPARK_HOME/bin/spark-submit --version
# Set Flink Home (if using Flink engine)
export FLINK_HOME=/path/to/flink
# Verify Flink installation
$FLINK_HOME/bin/flink --version
| Requirement | Version/Spec |
|---|---|
| Java | JDK 1.8+ |
| Memory | 2GB+ (minimum), 8GB+ (recommended) |
| Disk | 500MB (binary) + job storage |
| Network | Connectivity to source/sink systems |
| Scala | 2.12.15 (for Spark/Flink integration) |
SeaTunnel uses HOCON (Human-Optimized Config Object Notation) for job configuration.
Basic Structure:
env {
job.mode = "BATCH" # or STREAMING
job.name = "My Job"
parallelism = 4
}
source {
SourceConnector {
option1 = value1
option2 = value2
schema = {
fields {
column1 = "type"
column2 = "type"
}
}
}
}
# Optional: Transform data
transform {
TransformName {
option = value
}
}
sink {
SinkConnector {
option1 = value1
option2 = value2
}
}
config/mysql_to_postgres.conf
env {
job.mode = "BATCH"
job.name = "MySQL to PostgreSQL"
}
source {
Jdbc {
driver = "com.mysql.cj.jdbc.Driver"
url = "jdbc:mysql://mysql-host:3306/mydb"
user = "root"
password = "password"
query = "SELECT * FROM users"
connection_check_timeout_sec = 100
}
}
sink {
Jdbc {
driver = "org.postgresql.Driver"
url = "jdbc:postgresql://pg-host:5432/mydb"
user = "postgres"
password = "password"
database = "mydb"
table = "users"
primary_keys = ["id"]
connection_check_timeout_sec = 100
}
}
Run:
seatunnel.sh -c config/mysql_to_postgres.conf -e spark
config/kafka_to_es.conf
env {
job.mode = "STREAMING"
job.name = "Kafka to Elasticsearch"
parallelism = 2
}
source {
Kafka {
bootstrap.servers = "kafka-host:9092"
topic = "events"
patterns = "event.*"
consumer.group = "seatunnel-group"
format = "json"
schema = {
fields {
event_id = "bigint"
event_name = "string"
timestamp = "bigint"
payload = "string"
}
}
}
}
transform {
Sql {
sql = "SELECT event_id, event_name, FROM_UNIXTIME(timestamp/1000) as ts, payload FROM source"
}
}
sink {
Elasticsearch {
hosts = ["es-host:9200"]
index = "events"
index_type = "_doc"
username = "elastic"
password = "password"
}
}
Run:
seatunnel.sh -c config/kafka_to_es.conf -e flink
config/mysql_cdc_kafka.conf
env {
job.mode = "STREAMING"
job.name = "MySQL CDC to Kafka"
}
source {
Mysql {
server_id = 5400
hostname = "mysql-host"
port = 3306
username = "root"
password = "password"
database = ["mydb"]
table = ["users", "orders"]
startup.mode = "initial"
snapshot.split.size = 8096
incremental.snapshot.chunk.size = 1024
snapshot_fetch_size = 1024
snapshot_lock_timeout_sec = 10
server_time_zone = "UTC"
}
}
sink {
Kafka {
bootstrap.servers = "kafka-host:9092"
topic = "mysql_cdc"
format = "canal_json"
semantic = "EXACTLY_ONCE"
}
}
Run:
seatunnel.sh -c config/mysql_cdc_kafka.conf -e flink
# Using Spark (default)
seatunnel.sh -c config/job.conf -e spark
# Using Flink
seatunnel.sh -c config/job.conf -e flink
# Using Zeta (lightweight)
seatunnel.sh -c config/job.conf -e zeta
# Submit to Spark cluster
seatunnel.sh -c config/job.conf -e spark -m cluster -n hadoop-master:7077
# Submit to Flink cluster
seatunnel.sh -c config/job.conf -e flink -m remote -s localhost 8081
seatunnel.sh -c config/job.conf -e spark -l DEBUG
# View running jobs (Spark Cluster)
spark-submit --status <driver-id>
# View running jobs (Flink Cluster)
$FLINK_HOME/bin/flink list
Use SQL for complex transformations:
source {
MySQL {
# Source config...
}
}
transform {
Sql {
# Multiple SQL statements
sql = """
SELECT
id,
UPPER(name) as name,
age + 10 as age_plus_10,
CURRENT_TIMESTAMP as created_at
FROM source
WHERE age > 18
"""
}
}
sink {
PostgreSQL {
# Sink config...
}
}
All source connectors above also have corresponding sink implementations where applicable.
source {
ConnectorName {
# Connection
hostname = "host"
port = 3306
username = "user"
password = "pass"
# Data selection
database = "db_name"
table = "table_name"
# Performance
fetch_size = 1000
connection_check_timeout_sec = 100
split_size = 10000
# Schema
schema = {
fields {
id = "bigint"
name = "string"
}
}
}
}
sink {
ConnectorName {
# Connection
hostname = "host"
port = 3306
username = "user"
password = "pass"
# Write behavior
database = "db_name"
table = "table_name"
primary_keys = ["id"]
batch_size = 500
# Error handling
max_retries = 3
retry_wait_time_ms = 1000
on_duplicate_key_update_column_names = ["field1"]
}
}
env {
# Execution mode
job.mode = "BATCH" # or STREAMING
# Job identity
job.name = "My Job"
# Parallelism
parallelism = 4
# Checkpoint (streaming)
checkpoint.interval = 60000
# Restart strategy
restart_strategy = "fixed_delay"
restart_strategy.fixed_delay.attempts = 3
restart_strategy.fixed_delay.delay = 10000
}
# During execution, monitor logs
tail -f logs/seatunnel.log
# Check specific log level
grep ERROR logs/seatunnel.log
# Set log level to DEBUG
seatunnel.sh -c config/job.conf -e spark -l DEBUG
source {
FakeSource {
row.num = 1000
schema = {
fields {
id = "bigint"
name = "string"
}
}
}
}
sink {
Console {
format = "json"
}
}
seatunnel/
├── bin/ # Executable scripts
│ ├── seatunnel.sh # Main entry point
│ └── seatunnel-submit.sh # Spark submission script
├── config/ # Configuration examples
│ ├── flink-conf.yaml # Flink configuration
│ └── spark-conf.yaml # Spark configuration
├── connectors/ # Pre-built connectors
├── lib/ # JAR dependencies
├── logs/ # Runtime logs
└── plugin/ # Plugin directory
# Java configuration
export JAVA_HOME=/path/to/java
export JVM_OPTS="-Xms1G -Xmx4G"
# Spark configuration
export SPARK_HOME=/path/to/spark
export SPARK_MASTER=spark://master:7077
# Flink configuration
export FLINK_HOME=/path/to/flink
# SeaTunnel configuration
export SEATUNNEL_HOME=/path/to/seatunnel
seatunnel-env.shConfigure runtime environment:
JAVA_HOME=/path/to/java
JVM_OPTS="-Xms1G -Xmx4G -XX:+UseG1GC"
SPARK_HOME=/path/to/spark
FLINK_HOME=/path/to/flink
flink-conf.yaml (for Flink engine)taskmanager.memory.process.size: 2g
taskmanager.memory.jvm-overhead.fraction: 0.1
parallelism.default: 4
spark-conf.yaml (for Spark engine)driver-memory: 2g
executor-memory: 4g
num-executors: 4
env {
job.mode = "BATCH"
parallelism = 8 # Increase for larger clusters
}
source {
Jdbc {
# Split large tables for parallel reads
split_size = 100000
fetch_size = 5000
}
}
sink {
Jdbc {
# Batch inserts for better throughput
batch_size = 1000
# Use connection pooling
max_retries = 3
}
}
env {
job.mode = "STREAMING"
parallelism = 4
checkpoint.interval = 30000 # 30 seconds
}
source {
Kafka {
# Consumer group for parallel reads
consumer.group = "seatunnel-consumer"
# Batch reading
max_poll_records = 500
}
}
The official documentation site (https://seatunnel.apache.org) is managed through the apache/seatunnel-website repository, built with Docusaurus 2.4.3.
Documentation content lives in the main codebase (apache/seatunnel repo, docs/ directory). The website repo fetches docs via npm run sync (or tools/build-docs.js) before building.
The sync process:
swap/ directory, clones apache/seatunnel codebasedocs/en to website's docs/ directorydocs/zh to i18n/zh-CN/ for Chinese translationsdocs/images to static/image_en and static/image_zhdocs/sidebars.js to root levelseatunnel-website/
├── blog/ # Blog posts (EN)
├── community/ # Community docs
│ ├── contribution_guide/ # Contribution guidelines
│ │ ├── contribute.md
│ │ ├── committer.md
│ │ ├── code-review.md
│ │ ├── release.md
│ │ └── subscribe.md
│ └── submit_guide/ # Submission guidelines
│ ├── document.md
│ ├── license.md
│ └── submit-code.md
├── docs/ # Current version docs (synced from main repo)
├── docusaurus.config.js # Site configuration
├── i18n/zh-CN/ # Chinese translations
│ ├── docusaurus-plugin-content-blog/
│ ├── docusaurus-plugin-content-docs/
│ └── docusaurus-plugin-content-docs-community/
├── package.json
├── sidebars.js # Doc sidebar config (synced)
├── src/
│ ├── components/
│ ├── css/
│ ├── pages/
│ │ ├── home/ # Homepage
│ │ ├── team/ # Team page (with Base64 avatars)
│ │ ├── user/ # User showcase
│ │ └── versions/ # Version selector
│ └── styles/
├── static/ # Static assets
│ ├── doc/image/, image_en/, image_zh/
│ ├── home/, image/, user/
│ └── js/google_translate_init.js
├── tools/
│ ├── build-docs.js # Doc sync script
│ ├── common.js # Shared constants
│ ├── version.js # Version management
│ ├── image-copy.js # Image processing
│ └── fetch-team-avatars.js # Team avatar updater
├── versioned_docs/ # Historical version docs (20 versions: 1.x ~ 2.3.12)
├── versioned_sidebars/ # Historical sidebars
├── versions.json # Version registry
└── seatunnel_web_versions.json # Web UI version registry
Each version's docs contain:
about.md - Project overviewcommand/ - CLI commands (e.g., connector-check)concept/ - Core concepts (config, schema-evolution, speed-limit, SQL config, event-listener)connector-v2/ - Connector documentation
source/ - Source connector docs (100+ connectors)sink/ - Sink connector docschangelog/ - Per-connector changelogsformats/ - Data formats (Avro, Canal JSON, Debezium JSON, OGG JSON, Protobuf)contribution/ - Contributor guidesfaq.md - Frequently asked questionsother-engine/ - Flink/Spark engine specificsseatunnel-engine/ - Zeta engine docs + telemetrystart-v2/ - Getting started guides
docker/ - Docker deploymentkubernetes/ - K8s deploymentlocally/ - Local setuptransform-v2/ - Transform documentation (SQL, etc.)20 historical versions maintained: 1.x, 2.1.02.1.3, 2.2.0-beta, 2.3.0-beta2.3.12
Total: ~3691 versioned doc files + ~1652 i18n files
# Clone
git clone git@github.com:apache/seatunnel-website.git
cd seatunnel-website
# Sync docs from main repo
npm run sync
# Or with SSH: export PROTOCOL_MODE=ssh && npm run sync
# Install dependencies
npm install
# Dev server (English)
npm run start
# Dev server (Chinese)
npm run start-zh
# Production build (needs ~10GB heap)
npm run build
# Or faster parallel build
npm run build:fast
# Serve built site
npm run serve
# 1. Create versioned snapshot
npm run version <target_version>
# 2. Update download links
# Edit src/pages/download/st_data.json
| Branch | Purpose |
|---|---|
main | Default development branch |
asf-site | Production (https://seatunnel.apache.org) |
asf-staging | Staging (https://seatunnel.staged.apache.org) |
GitHub Actions workflow (.github/workflows/deploy.yml):
main, PRs to main, daily cron (5:00 AM UTC)npm install -> npm run sync -> npm run build -> deploy to asf-site branchscripts, components)render-dom.js)static/{module_name}src/css/SeaTunnel follows a modular architecture:
seatunnel/
├── seatunnel-api/ # Core APIs
├── seatunnel-core/ # Execution engine
├── seatunnel-engines/ # Engine implementations
│ ├── seatunnel-engine-flink/
│ ├── seatunnel-engine-spark/
│ └── seatunnel-engine-zeta/
├── seatunnel-connectors/ # 100+ connector implementations
│ ├── seatunnel-connectors-*/ # One per connector type
└── seatunnel-dist/ # Distribution package
# Clone repository
git clone https://github.com/apache/seatunnel.git
cd seatunnel
# Build all modules
mvn clean install -DskipTests
# Build with tests (slower)
mvn clean install
# Build only Kafka connector
mvn clean install -pl seatunnel-connectors/seatunnel-connectors-seatunnel-kafka -DskipTests
# Build only Flink engine
mvn clean install -pl seatunnel-engines/seatunnel-engine-flink -DskipTests
# Create binary distribution
mvn clean install -DskipTests
cd seatunnel-dist
tar -tzf target/apache-seatunnel-*-bin.tar.gz | head
# Run all tests
mvn test
# Run specific test class
mvn test -Dtest=MySqlConnectorTest
# Run with coverage
mvn test jacoco:report
# Run integration tests
mvn verify
# Run with Docker containers
mvn verify -Pintegration-tests
Import Project
Configure JDK
Enable Annotation Processing
Run Configuration
org.apache.seatunnel.core.starter.command.CommandExecuteRunnerSeaTunnel follows Apache project conventions:
Use the provided .editorconfig:
# Install EditorConfig plugin (IntelliJ)
# Then your IDE will auto-format code
import org.apache.seatunnel.api.source.SeSource;
import org.apache.seatunnel.api.table.catalog.Table;
public class CustomSource extends SeSource {
@Override
public String getPluginName() {
return "Custom";
}
@Override
public void validate() {
// Validation logic
}
@Override
public ResultSet read(Boundedness boundedness) {
// Implementation
}
}
import org.apache.seatunnel.api.configuration.Option;
import org.apache.seatunnel.api.configuration.Options;
public class CustomSourceOptions {
public static final Option<String> HOST =
Options.key("host")
.stringType()
.noDefaultValue()
.withDescription("Source hostname");
public static final Option<Integer> PORT =
Options.key("port")
.intType()
.defaultValue(9200)
.withDescription("Source port");
}
META-INF/services/org.apache.seatunnel.api.source.SeSource
Fork and Clone
git clone https://github.com/YOUR_USERNAME/seatunnel.git
cd seatunnel
git remote add upstream https://github.com/apache/seatunnel.git
Create Feature Branch
git checkout -b feature/my-feature
Make Changes
Commit and Push
git add .
git commit -m "feat: add new feature"
git push origin feature/my-feature
Create Pull Request
Code Review Process
Cause: JDBC driver JAR not in classpath
Solution:
# 1. Download MySQL JDBC driver
wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-8.0.33.jar
# 2. Copy to lib directory
cp mysql-connector-java-8.0.33.jar $SEATUNNEL_HOME/lib/
# 3. Restart job
seatunnel.sh -c config/job.conf -e spark
Cause: Insufficient JVM heap memory
Solution:
# Increase JVM memory
export JVM_OPTS="-Xms2G -Xmx8G"
# Or set in seatunnel-env.sh
echo 'JVM_OPTS="-Xms2G -Xmx8G"' >> $SEATUNNEL_HOME/bin/seatunnel-env.sh
Cause: Source/sink service not reachable
Solution:
# 1. Verify connectivity
ping source-host
telnet source-host 3306
# 2. Check credentials
mysql -h source-host -u root -p
# 3. Check firewall rules
# Ensure port 3306 is open
Cause: Binlog not enabled on MySQL
Solution:
-- Check binlog status
SHOW VARIABLES LIKE 'log_bin';
-- Enable binlog in my.cnf
[mysqld]
log_bin = mysql-bin
binlog_format = row # Important for CDC
-- Restart MySQL and verify
SHOW MASTER STATUS;
Cause: Suboptimal configuration
Solutions:
# 1. Increase parallelism
env {
parallelism = 8 # or higher based on cluster
}
# 2. Increase batch sizes
source {
Jdbc {
fetch_size = 5000
split_size = 100000
}
}
sink {
Jdbc {
batch_size = 2000
}
}
# 3. Enable connection pooling
source {
Jdbc {
pool_size = 10
max_idle_time = 300
}
}
Cause: Offset doesn't exist in topic
Solution:
source {
Kafka {
# Reset to earliest or latest
auto.offset.reset = "earliest" # or "latest"
# Or specify explicit offsets
start_mode = "earliest"
}
}
Q: What's the difference between BATCH and STREAMING mode?
A:
Q: How do I handle schema changes during CDC?
A: SeaTunnel automatically detects schema changes in CDC mode. Configure in source:
source {
Mysql {
schema_change_mode = "auto" # auto-detect and apply
}
}
Q: Can I transform data during synchronization?
A: Yes, use SQL transform:
transform {
Sql {
sql = "SELECT id, UPPER(name) as name FROM source"
}
}
Q: What's the maximum throughput?
A: Depends on:
Typical throughput: 100K - 1M records/second per executor
Q: How do I handle errors in production?
A: Configure restart strategy:
env {
restart_strategy = "exponential_delay"
restart_strategy.exponential_delay.initial_delay = 1000
restart_strategy.exponential_delay.max_delay = 30000
restart_strategy.exponential_delay.multiplier = 2.0
restart_strategy.exponential_delay.attempts_unlimited = true
}
Q: Is there a web UI for job management?
A: Yes! Use SeaTunnel Web Project:
# Check out web UI project
git clone https://github.com/apache/seatunnel-web.git
cd seatunnel-web
mvn clean install
# Run web UI
java -jar target/seatunnel-web-*.jar
# Access at http://localhost:8080
Apache License 2.0 - See LICENSE file
Last Updated: 2026-02-26 Skill Version: 2.3.13 Sources: apache/seatunnel + apache/seatunnel-website