| name | syn-spark-checkpointing |
| description | Operate, configure, troubleshoot, and manage Apache Spark. Use for Apache Spark engineering tasks, day-to-day operations, and production support. |
Apache Spark: checkpointing and lineage truncation
When to use
Use this skill for checkpointing and lineage truncation in Apache Spark. This skill
specifically handles checkpointing and lineage truncation and nothing else in the
Apache Spark family.
Procedure
Use checkpoint() to truncate long lineage in iterative jobs, configure the checkpoint dir, and balance cost vs recompute.
Notes
This is the Apache Spark skill dedicated to checkpointing and lineage truncation.
Sibling skills cover other Apache Spark capabilities; this one
is the right choice only when the task is about checkpointing and lineage truncation.