newfront

Work with Delta Lake tables in PySpark — SparkSession configuration, reading and writing Delta tables, schema evolution, MERGE/upsert, time travel, OPTIMIZE, VACUUM, partitioning, liquid clustering, batched writes, and testing patterns. Use when creating, reading, or writing Delta tables, configuring Delta in Spark, performing upserts/merges, managing table maintenance, or asking about Delta Lake best practices in PySpark.

Map protobuf-sourced clickstream data (the Tidewell GA4 event/user/product schema from pyspark-datagen) to the correct Spark column paths in an SDP pipeline. Use this skill whenever decoding `event.v1.Event` with `from_protobuf`, projecting fields out of a protobuf `oneof` payload, reconstructing money from a `commerce.v1.Amount` (units + nanos), exploding `items[]` line items, or whenever the user mentions protobuf, descriptors, GA4 events, sessionization, the buy-flow funnel, or the events/users/products datasets. Reach for this before writing any `F.col(...)` against decoded protobuf data — the field paths are non-obvious and easy to hallucinate as flat columns.

Write fast, deterministic PySpark unit tests with pytest. Use when adding or editing tests for PySpark DataFrame transformations, setting up a shared SparkSession fixture, or scaffolding a new test module under `tests/`.

Author Spark Declarative Pipelines (SDP) the way this repo expects — pure logic in `lib/`, decorated entry points in `transformations/`, reads inside the flow body, no actions in dataset code. Use this skill whenever adding or editing a `@dp.table` / `@dp.materialized_view` / `@dp.temporary_view` / `@dp.append_flow`, wiring a new dataset into `example_pipeline/`, debugging "graph element defined outside of declarative pipeline" or "session mutation" errors, or whenever the user mentions SDP, declarative pipelines, dataflow graphs, streaming tables, materialized views, `spark-pipelines`, or `just sdp-run` / `just dry-run`.

Work with Delta Lake tables in PySpark — SparkSession configuration, reading and writing Delta tables, schema evolution, MERGE/upsert, time travel, OPTIMIZE, VACUUM, partitioning, liquid clustering, batched writes, and testing patterns. Use when creating, reading, or writing Delta tables, configuring Delta in Spark, performing upserts/merges, managing table maintenance, or asking about Delta Lake best practices in PySpark.

Use Protobuf messages with Apache Spark and Delta Lake — SparkSession setup, building and loading FileDescriptorSet files, deserializing binary protobuf into DataFrames via from_protobuf, and writing to Delta. Use when working with protobuf in PySpark, converting proto bytes to DataFrames, reading descriptor files, or ingesting protobuf data into Delta Lake.

Fetches, displays, and summarizes GitHub issues from any accessible repository. Supports filtering by limit, state, and optionally expanding issue bodies with AI summarization, and fetching related open PRs and branches per issue. Use when the user asks to fetch issues, list GitHub issues, summarize issues, find bugs or feature requests in a repo, or says things like "gh issues", "get issues from", "show me issues for", or provides a GitHub repo URL and asks about its issues.

Fetches GitHub issues and performs hybrid sentiment analysis using VADER (offline scoring) and the Cursor agent as the LLM (no API key required). Produces a sentiment-enriched JSON or markdown report per issue: tone label, tone detail, and a one-sentence summary. Use when the user asks to analyze sentiment of GitHub issues, score issue tone, classify issue mood, summarize issues with sentiment, or says things like "analyze sentiment", "score these issues", "what's the tone of this issue", or "run sentiment analysis without an API key".

Add a new Protobuf-backed data generator to the learning-spark-datagen project (new proto, GenX class, CLI, tests). Use when adding a new entity type (e.g. Shipment, Payment), a new message with deterministic fake data, or when the user asks to create a generator following the User/Order pattern.