spark-development

Stars0

Forks0

UpdatedJune 10, 2026 at 12:49

Write and optimize PySpark and Spark SQL. Use for Spark DataFrames, joins, windows, UDF decisions, partitioning, caching, AQE, explain plans, OOM, shuffle, skew, Delta writes, and Spark pipeline performance.

Installation

Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.

Source

datacoolie

datacoolie/dekit

View GitHub Repository View Creator Repositories

Download

Related occupationsSOC

Based on SOC occupation classification

Software DevelopersComputer and Mathematical Occupations·SOC 15-1252

SKILL.md

readonly

name	spark-development
description	Write and optimize PySpark and Spark SQL. Use for Spark DataFrames, joins, windows, UDF decisions, partitioning, caching, AQE, explain plans, OOM, shuffle, skew, Delta writes, and Spark pipeline performance.

Spark Development

Use this skill for Spark implementation choices and performance risk checks.

Rules

Define schemas explicitly at production read boundaries.
Prefer built-in functions over pandas UDFs; pandas UDFs over Python UDFs.
Keep transformations set-based; avoid driver-side loops.
Use idempotent writes: MERGE/upsert, replaceWhere, or partition overwrite.
Partition by query/lifecycle pattern, not convenience.
Cache only DataFrames reused across multiple actions; unpersist after use.
Inspect plans before optimizing: explain("formatted") or platform equivalent.
Tune from observed data size, skew, shuffle, and SLA, not guesses.

Join Checklist

Expected cardinality is known before joining.
Join keys have compatible types and null behavior.
Broadcast only genuinely small sides.
No cartesian product unless deliberately bounded.
For skew, validate key distribution before salting or repartitioning.

Red Flags

collect(), toPandas(), or repartition(1) on large data.
inferSchema=True in production.
Python UDF for simple expressions.
Repeated .count() actions just for logging.
Writes without atomicity or retry/idempotency plan.
Partition columns missing from common filters.

Verification

Before reporting Spark work complete:

Run affected transformation or representative local/unit test.
Validate output schema and row counts.
Confirm rerun behavior is idempotent.
Inspect explain plan for unexpected cartesian joins, excessive exchanges, or missing pruning.
Reconcile critical measures against source or prior layer.

More from this repository

same repository

wiki

datacoolie/dekit

Build and maintain an internal technical LLM wiki when the user requests wiki work or a verified major or architectural change qualifies for automatic wiki review. Use for project memory, architecture notes, contracts, runbooks, decisions, glossary, incremental ingest, delta status, and wiki health. Do not use merely because a routine task or plan completed.

2026-07-180

brainstorm

datacoolie/dekit

Explore ambiguous ideas and solution directions before evidence-backed research or implementation planning. Use for ideation, framing, assumption checks, option generation, product or architecture trade-offs, and deciding whether research or a plan is needed.

2026-07-030

research

datacoolie/dekit

Research technical options with source-backed recommendations. Use for technology evaluation, architecture trade-offs, library/framework choices, best practices, scalability, security, maintainability, or when current external facts matter.

2026-07-030

plan

datacoolie/dekit

Create implementation plans, architecture decisions, and phased roadmaps. Use for Standard or Complex changes, new pipelines, schema changes, migrations, multi-file features, technology choices, or when acceptance criteria and verification strategy are unclear.

2026-06-170

code-review

datacoolie/dekit

Review code and data pipeline changes for correctness, security, performance, maintainability, contract breaks, and missing verification. Use before merge, after implementation, for PRs, commits, pending diffs, or codebase risk scans.

2026-06-100

data-ingestion

datacoolie/dekit

Design data ingestion into landing/bronze layers. Use for source onboarding, transfer pattern selection, full/incremental/CDC scope, change detection, landing zones, schema drift, idempotent file or batch processing, and ingestion quality gates.

2026-06-100

name	spark-development
description	Write and optimize PySpark and Spark SQL. Use for Spark DataFrames, joins, windows, UDF decisions, partitioning, caching, AQE, explain plans, OOM, shuffle, skew, Delta writes, and Spark pipeline performance.

Spark Development

Use this skill for Spark implementation choices and performance risk checks.

Rules

Define schemas explicitly at production read boundaries.
Prefer built-in functions over pandas UDFs; pandas UDFs over Python UDFs.
Keep transformations set-based; avoid driver-side loops.
Use idempotent writes: MERGE/upsert, replaceWhere, or partition overwrite.
Partition by query/lifecycle pattern, not convenience.
Cache only DataFrames reused across multiple actions; unpersist after use.
Inspect plans before optimizing: explain("formatted") or platform equivalent.
Tune from observed data size, skew, shuffle, and SLA, not guesses.

Join Checklist

Expected cardinality is known before joining.
Join keys have compatible types and null behavior.
Broadcast only genuinely small sides.
No cartesian product unless deliberately bounded.
For skew, validate key distribution before salting or repartitioning.

Red Flags

collect(), toPandas(), or repartition(1) on large data.
inferSchema=True in production.
Python UDF for simple expressions.
Repeated .count() actions just for logging.
Writes without atomicity or retry/idempotency plan.
Partition columns missing from common filters.

Verification

Before reporting Spark work complete:

Run affected transformation or representative local/unit test.
Validate output schema and row counts.
Confirm rerun behavior is idempotent.
Inspect explain plan for unexpected cartesian joins, excessive exchanges, or missing pruning.
Reconcile critical measures against source or prior layer.