Run any Skill in Manus with one click

paimon

Stars31

Forks3

UpdatedJune 2, 2026 at 19:57

Apache Paimon streaming lake format expertise for real-time ingestion and lakehouse architectures. Use when the user mentions Paimon, streaming lakehouse, primary-key tables, changelog, Flink CDC into a lake, Paimon Materialized Tables, PyPaimon, deletion vectors, lookup joins, buckets, compaction, or Spark/Flink Paimon tables.

Installation

Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.

Run Skill in Manus

Source

gordonmurray

gordonmurray/data-engineering-skills

View GitHub Repository View Creator Repositories

Download

Run Skill in Manus

Related occupationsSOC

Based on SOC occupation classification

Software DevelopersComputer and Mathematical Occupations·SOC 15-1252

SKILL.md

readonly

name	paimon
description	Apache Paimon streaming lake format expertise for real-time ingestion and lakehouse architectures. Use when the user mentions Paimon, streaming lakehouse, primary-key tables, changelog, Flink CDC into a lake, Paimon Materialized Tables, PyPaimon, deletion vectors, lookup joins, buckets, compaction, or Spark/Flink Paimon tables.

Apache Paimon Expert

Use this skill for Paimon table design, Flink-native streaming ingestion, changelog semantics, compaction, lookup joins, Spark reads, and Iceberg compatibility.

Current Facts

Current stable Paimon: 1.4.1. A 1.4.2 release candidate exists; do not recommend it as stable unless the user explicitly wants RC testing.
PyPaimon: 1.4.1 on PyPI, pure Python package.
Flink CDC: 3.6.0 is the current CDC line; older 3.5 examples remain useful but should not be described as latest.
Recommended Flink: Flink 1.20.x or 2.2.x for new work when connector compatibility allows.
Recommended Spark: verify against the Paimon connector matrix for the selected Paimon version; do not hard-code Spark 3.4.3 for new projects without checking.
Recent focus areas: PyPaimon, data evolution, Iceberg compatibility, deletion vectors, REST Catalog authorization interfaces, lookup join performance, multimodal/blob storage, and Paimon/Lance integration work.

How To Use

Determine table type first: append-only table or primary-key table.
Determine workload: streaming ingest, CDC upsert, lookup dimension table, batch analytics, or cross-format Iceberg exposure.
Choose bucket strategy early; bucket count affects write parallelism, small files, and lookup performance.

Design Rules

Use primary-key tables for upserts, deletes, and CDC; use append-only tables for immutable event logs.
Include partition fields in primary keys when tables are partitioned.
Avoid single-bucket defaults for large tables; choose fixed or dynamic buckets deliberately.
Use changelog producer settings based on downstream needs: input, lookup, full-compaction, or none.
Plan compaction separately from ingestion for high-volume streaming tables.
Use lookup cache only when dimension-table size and freshness requirements justify it.

Update Checklist

Recheck Apache Paimon tags/downloads and PyPI pypaimon before changing versions.
Recheck Flink CDC compatibility for the selected Flink and Paimon releases.

More from this repository

same repository

docker-compose

gordonmurray/data-engineering-skills

Docker Compose V2 and Compose Specification expertise for writing correct compose.yaml and docker-compose.yml files. Use when the user mentions Docker Compose, compose.yaml, docker-compose.yml, docker compose CLI, multi-container apps, profiles, healthchecks, depends_on, networks, volumes, secrets, build contexts, or avoiding deprecated V1 patterns.

2026-06-0231

flink

gordonmurray/data-engineering-skills

Apache Flink stream processing expertise for stateful computations over bounded and unbounded streams. Use when the user mentions Flink, Flink SQL, Table API, DataStream API, CDC, Kafka-to-Flink pipelines, checkpoints, savepoints, watermarks, windows, Flink Kubernetes Operator, or Flink integrations with Iceberg, Paimon, or Fluss.

2026-06-0231

fluss

gordonmurray/data-engineering-skills

Apache Fluss Incubating streaming storage expertise for real-time analytics, streamhouse and lakehouse architectures. Use when the user mentions Fluss, streaming storage, columnar streams, sub-second ingestion, tiered storage, Flink Delta Join, Paimon/Iceberg/Lance tiering, Spark access to streams, or Fluss table design.

2026-06-0231

iceberg

gordonmurray/data-engineering-skills

Apache Iceberg table format expertise for data lakehouses. Use when the user mentions Iceberg, lakehouse, table format v2 or v3, deletion vectors, partition evolution, time travel, snapshot isolation, Polaris, Nessie, REST catalogs, metadata.json, manifest lists, snapshots, Spark, Flink, Trino, Athena, Snowflake, or table maintenance.

2026-06-0231

iggy

gordonmurray/data-engineering-skills

Apache Iggy Incubating Rust-native message streaming platform expertise. Use when the user mentions Iggy, event streaming, Kafka/NATS alternatives, Iggy CLI or SDKs, Python apache-iggy, Docker apache/iggy images, Iggy MCP, QUIC/TCP/HTTP/WebSocket transports, streams, topics, partitions, retention, or consumer groups.

2026-06-0231

lance

gordonmurray/data-engineering-skills

Lance columnar data format and LanceDB expertise for ML/AI workloads, vector search, embeddings, multimodal datasets, and .lance files. Use when the user mentions Lance, LanceDB, pylance, vector indexes, ANN search, IVF_PQ, IVF_HNSW_FLAT, HNSW, full-text search, dataset versioning, or migrating ML data from Parquet.

2026-06-0231

name	paimon
description	Apache Paimon streaming lake format expertise for real-time ingestion and lakehouse architectures. Use when the user mentions Paimon, streaming lakehouse, primary-key tables, changelog, Flink CDC into a lake, Paimon Materialized Tables, PyPaimon, deletion vectors, lookup joins, buckets, compaction, or Spark/Flink Paimon tables.

Apache Paimon Expert

Use this skill for Paimon table design, Flink-native streaming ingestion, changelog semantics, compaction, lookup joins, Spark reads, and Iceberg compatibility.

Current Facts

Current stable Paimon: 1.4.1. A 1.4.2 release candidate exists; do not recommend it as stable unless the user explicitly wants RC testing.
PyPaimon: 1.4.1 on PyPI, pure Python package.
Flink CDC: 3.6.0 is the current CDC line; older 3.5 examples remain useful but should not be described as latest.
Recommended Flink: Flink 1.20.x or 2.2.x for new work when connector compatibility allows.
Recommended Spark: verify against the Paimon connector matrix for the selected Paimon version; do not hard-code Spark 3.4.3 for new projects without checking.
Recent focus areas: PyPaimon, data evolution, Iceberg compatibility, deletion vectors, REST Catalog authorization interfaces, lookup join performance, multimodal/blob storage, and Paimon/Lance integration work.

How To Use

Determine table type first: append-only table or primary-key table.
Determine workload: streaming ingest, CDC upsert, lookup dimension table, batch analytics, or cross-format Iceberg exposure.
Choose bucket strategy early; bucket count affects write parallelism, small files, and lookup performance.

Design Rules

Use primary-key tables for upserts, deletes, and CDC; use append-only tables for immutable event logs.
Include partition fields in primary keys when tables are partitioned.
Avoid single-bucket defaults for large tables; choose fixed or dynamic buckets deliberately.
Use changelog producer settings based on downstream needs: input, lookup, full-compaction, or none.
Plan compaction separately from ingestion for high-volume streaming tables.
Use lookup cache only when dimension-table size and freshness requirements justify it.

Update Checklist

Recheck Apache Paimon tags/downloads and PyPI pypaimon before changing versions.
Recheck Flink CDC compatibility for the selected Flink and Paimon releases.