learning-spark-sdp

Work with Delta Lake tables in PySpark — SparkSession configuration, reading and writing Delta tables, schema evolution, MERGE/upsert, time travel, OPTIMIZE, VACUUM, partitioning, liquid clustering, batched writes, and testing patterns. Use when creating, reading, or writing Delta tables, configuring Delta in Spark, performing upserts/merges, managing table maintenance, or asking about Delta Lake best practices in PySpark.

Map protobuf-sourced clickstream data (the Tidewell GA4 event/user/product schema from pyspark-datagen) to the correct Spark column paths in an SDP pipeline. Use this skill whenever decoding `event.v1.Event` with `from_protobuf`, projecting fields out of a protobuf `oneof` payload, reconstructing money from a `commerce.v1.Amount` (units + nanos), exploding `items[]` line items, or whenever the user mentions protobuf, descriptors, GA4 events, sessionization, the buy-flow funnel, or the events/users/products datasets. Reach for this before writing any `F.col(...)` against decoded protobuf data — the field paths are non-obvious and easy to hallucinate as flat columns.

Write fast, deterministic PySpark unit tests with pytest. Use when adding or editing tests for PySpark DataFrame transformations, setting up a shared SparkSession fixture, or scaffolding a new test module under `tests/`.

Author Spark Declarative Pipelines (SDP) the way this repo expects — pure logic in `lib/`, decorated entry points in `transformations/`, reads inside the flow body, no actions in dataset code. Use this skill whenever adding or editing a `@dp.table` / `@dp.materialized_view` / `@dp.temporary_view` / `@dp.append_flow`, wiring a new dataset into `example_pipeline/`, debugging "graph element defined outside of declarative pipeline" or "session mutation" errors, or whenever the user mentions SDP, declarative pipelines, dataflow graphs, streaming tables, materialized views, `spark-pipelines`, or `just sdp-run` / `just dry-run`.