skill-pyspark-expert

Write production-quality PySpark code, unit tests, and project layouts for Spark 3.5+ with Delta Lake. Use this skill whenever the user asks to write, review, refactor, or debug PySpark code, create pytest-based unit tests for Spark transformations, define schemas, design ETL modules, or discuss DataFrame API best practices. Also trigger when the user mentions PySpark, Spark DataFrame, SparkSession, pyspark.sql.functions, Delta Lake tables, or asks for help structuring a Spark project. Even if the request seems simple (e.g., "write a groupBy"), consult this skill because it enforces code style, anti-pattern avoidance, and idiomatic Spark 3.5+ usage that Claude would otherwise miss.