lakeflow_declarative_pipelines_demo

Creates, configures, and updates Databricks Lakeflow Spark Declarative Pipelines (SDP/LDP) using serverless compute. Handles data ingestion with streaming tables, materialized views, CDC, SCD Type 2, and Auto Loader ingestion patterns. Use when building data pipelines, working with Delta Live Tables, ingesting streaming data, implementing change data capture, or when the user mentions SDP, LDP, DLT, Lakeflow pipelines, streaming tables, or bronze/silver/gold medallion architectures.

Write, structure, and maintain high-quality pytest test suites following official pytest documentation best practices.

Write clean, testable, and maintainable Python code using functional programming principles — pure functions, immutability, higher-order functions, and composable pipelines.

Write idiomatic, readable Python code following PEP 8 – the official Python style guide.

Write idiomatic, performant PySpark code following the Palantir PySpark Style Guide.