pyspark-pipeline-composer

Work with Delta Lake tables in PySpark — SparkSession configuration, reading and writing Delta tables, schema evolution, MERGE/upsert, time travel, OPTIMIZE, VACUUM, partitioning, liquid clustering, batched writes, and testing patterns. Use when creating, reading, or writing Delta tables, configuring Delta in Spark, performing upserts/merges, managing table maintenance, or asking about Delta Lake best practices in PySpark.

Use Protobuf messages with Apache Spark and Delta Lake — SparkSession setup, building and loading FileDescriptorSet files, deserializing binary protobuf into DataFrames via from_protobuf, and writing to Delta. Use when working with protobuf in PySpark, converting proto bytes to DataFrames, reading descriptor files, or ingesting protobuf data into Delta Lake.