flink

name	flink
description	Apache Flink stream processing expertise for stateful computations over bounded and unbounded streams. Use when the user mentions Flink, Flink SQL, Table API, DataStream API, CDC, Kafka-to-Flink pipelines, checkpoints, savepoints, watermarks, windows, Flink Kubernetes Operator, or Flink integrations with Iceberg, Paimon, or Fluss.
metadata	{"category":"data","source":{"repository":"https://github.com/gordonmurray/data-engineering-skills","path":"flink","license_path":"LICENSE","commit":"226973561ae6881d21b6567171195e8211863d75"}}

Apache Flink Data Streaming Expert

Use this skill for production Flink architecture, operations, SQL/DataStream implementation, upgrade planning, and lakehouse streaming integrations.

Current Facts

• Current Flink line: 2.2.x. Downloads include Apache Flink 2.2.1, while the site still labels 2.2.0 as the latest stable line.
• Maintained 2.x patch lines: 2.2.1, 2.1.2, 2.0.2.
• 1.x maintenance line: 1.20.4. Use this as the 1.x migration baseline unless the project is pinned elsewhere.
• Kubernetes Operator: 1.15.0, compatible with Flink 2.2.x, 2.1.x, 2.0.x, 1.20.x, and 1.19.x.
• Flink CDC: 3.6.0, with artifacts for Flink 1.20.x and 2.2.x.
• Java: Flink 2.x requires Java 11+; Java 17 is the practical default for new deployments.

Critical 2.x Notes

• DataSet API removed; use DataStream, Table API, or SQL.
• Scala DataStream/DataSet APIs removed from the core distribution.
• SourceFunction/SinkFunction and Sink V1 patterns are obsolete; prefer Source/Sink V2 connectors.
• flink-conf.yaml was replaced by standard YAML config.yaml in Flink 2.x.
• Per-job deployment mode was removed; use Application mode or Kubernetes Operator patterns.
• Validate savepoint compatibility carefully before 1.x to 2.x migrations.

How To Use

1. Classify the request: SQL/Table API, DataStream, deployment, operations, upgrade, CDC, or lakehouse sink/source.
2. For new greenfield work, prefer Flink 2.2.x plus current connector artifacts.
3. For migration work, identify the exact source version, connector versions, state backend, and savepoint strategy before recommending commands.

Production Defaults

• Enable checkpointing for streaming jobs and set explicit checkpoint storage.
• Use savepoints for planned upgrades and topology changes.
• Use watermarks and allowed lateness deliberately; make event-time assumptions visible.
• Monitor checkpoint duration, alignment time, backpressure, restart count, and state size.
• Prefer the Kubernetes Operator for long-running production jobs on Kubernetes.
• Use Iceberg/Paimon/Fluss connectors only at versions compatible with the selected Flink line.

Update Checklist

• Recheck Flink downloads for core, CDC, connector, and Kubernetes Operator versions.
• Update Helm/doc URLs when operator versions change.