Knowledge base from "Learning Spark, 2nd Edition" by Damji, Wenig, Das & Lee. Use when working with Apache Spark / PySpark, preparing for the Databricks Certified Associate Developer for Apache Spark exam, or applying the authors' frameworks for Spark architecture, DataFrame API, Spark SQL, Structured Streaming, tuning, and Datasets.
Knowledge base for Pandas API on Spark (pyspark.pandas) — Apache Spark 3.5. Use when preparing for Sec 7 objective 'Explain advantages of using Pandas API on Spark', migrating pandas code to scale on Spark, or configuring pyspark.pandas options (default index type, ops_on_diff_frames, checkpointing). Sources: spark.apache.org 3.5.7 docs + Databricks blog (Koalas merger). NOT covered here: Pandas UDF (separate skill), Spark Connect (see skill spark-connect).
Knowledge base for Spark 3.1→3.5 updates not covered by the Learning Spark 2nd Ed book. Use for: (A) Pandas UDF and Arrow APIs (exam Sec 7 obj 2 'Create and invoke Pandas UDF'); (B) new built-in functions added in 3.3-3.5 (exam Sec 3); (C) AQE evolution post-3.0 — default-on, new sub-configs, SMJ→SHJ conversion (exam Sec 4). NOT covered here: Pandas API on Spark (see skill pandas-on-spark), Spark Connect (see skill spark-connect), AQE 3.0 baseline (see apache-spark ch07/ch12).
Meta-skill orchestrator for preparing the Databricks Certified Associate Developer for Apache Spark 3.5 exam (Python). Coordinates the sub-skills apache-spark, spark-connect, pandas-on-spark, and spark-3.5-updates. Activates on requests like "let's study Spark", "review <topic>", "quiz on <topic>", "mock exam", "what do I still need to study", "weak spots", "progress check".
Knowledge base for Spark Connect (Apache Spark 3.4–3.5). Use when preparing for Sec 6 of the Databricks Certified Associate Developer for Apache Spark exam, configuring a Spark Connect client/server, or migrating PySpark code to the decoupled client-server architecture. Sources: spark.apache.org 3.5.7 docs + Databricks engineering blog.