open-lakehouse

MLflow 3.13 (tracking + AI Gateway) on this stack. Load when logging experiments from Spark/Python jobs, registering models, or wiring AI Gateway routes to Anthropic / Ollama for LLM-in-pipeline patterns.

Spark Declarative Pipelines (SDP) — the declarative pipeline framework in Spark 4.x. Load when designing, running, or debugging an SDP pipeline. Covers the OSS `pyspark.pipelines` API, dataset primitives, the `spark-pipelines` CLI, and the medallion pattern on this stack.

Delta Lake 4.2 operations on Spark 4.1 in this stack. Load when working with Delta tables specifically — covers OPTIMIZE, VACUUM, time travel, UniForm interop with Iceberg, and when to choose Delta over Iceberg here.

Kafka 3.6 producer/consumer patterns for this stack. Load when wiring Kafka into a Spark job, debugging stream lag, or writing demo producers. Covers topic creation, Structured Streaming integration, schema serialization, and the local test harness.

Apache Spark 4.1 reference. Load when writing PySpark/Spark SQL against this stack — covers DataFrame API conventions, Spark 4.1-specific features (Connect, ANSI mode, streaming UDTFs), and gotchas vs older Spark.

Unity Catalog OSS 0.4.x — the only catalog in this stack. Load when configuring the UC server, creating catalogs/schemas/tables via REST, or wiring a non-Spark engine (DuckDB, Trino) against UC. Covers the REST API surface, credential vending, and the no-JDBC-catalog rule.

Use when starting, stopping, demoing, or troubleshooting the open-lakehouse stack (Spark 4.1, Kafka, Airflow, Unity Catalog, MLflow). Provides deterministic start/stop/demo runbooks an AI agent can execute top-to-bottom.

Apache Airflow 3.1 orchestration for this stack. Load when writing DAGs, dealing with the 2→3 migration gotchas, configuring Spark/Kafka connections, or debugging Airflow inside the docker-compose-airflow.yml setup.

This skill should be used when preparing or polishing a release driven by release-plz (or release-please) — when the user wants to "prep the release", "review the release PR", "augment the release notes", "add highlights to the GitHub release", or otherwise turn the mechanical auto-generated changelog into a cleanly-reviewed, high-quality release. Two modes — pre-release prep (analyze the open Release PR, cross-check commits, find doc drift, open a blocking doc-fix PR, draft highlights) and post-release augmentation (prepend curated highlights to the published GitHub Releases).

Build proto-driven REST services with the Trestle/olai-codegen Rust framework. Use when writing or editing .proto files for a Trestle project, adding endpoints or resources, running `trestle generate` / `just regen`, implementing generated Axum handler traits, wiring generated clients, or wiring the olai-store resource layer. Covers the google.api.resource / google.api.http / google.api.field_behavior / google.api.resource_reference / debug_redact annotations and exactly what code each one generates.

This skill should be used when the user asks to "commit", "prepare a commit", "stage and commit", "commit my changes", or says the work is done and changes should be committed. Commits unsigned (no interactive GPG PIN) and defers signing to a single bulk step before opening a PR.

Classifies Unity Catalog tables and columns with governance tags (pii, phi, financial, clinical, public) and emits the right INSERT / MERGE / tags.json edit plus a matching Cedar template when no policy yet covers the tag. Use when the user asks to tag a column, tag a table, classify PII/PHI columns, add an entry to governance.policast.tags, or asks "what tag should this column have?" — i.e. the policast-cel tag-driven FGAC authoring flow from the data-owner's perspective.

Generates Cedar governance policies through a short Q&A interview, then emits the exact artifacts needed for policast-cel (Cedar source, manifest update command, and binding/tag changes). Use when the user asks for a policy wizard, policy questionnaire, “ask me questions and generate policy”, or needs help drafting row filters/column masks/deny overrides from requirements.

This skill should be used when the user asks to "commit", "prepare a commit", "stage and commit", "commit my changes", or says the work is done and changes should be committed. Commits unsigned (no interactive GPG PIN) and defers signing to a single bulk step before opening a PR.

Regenerate this site's machine-readable discovery files for SEO/GEO/AEO — public/sitemap.xml, public/llms.txt, and public/llms-full.txt — from the content in src/content. Use after adding or editing blog/learn posts, authors, routes, or tech categories, when the canonical domain changes, or whenever the user mentions sitemap, llms.txt, robots.txt, canonical URLs, SEO, GEO, or AEO for this repo.

This skill should be used when the user asks to "commit", "prepare a commit", "stage and commit", "commit my changes", or says the work is done and changes should be committed. Commits unsigned (no interactive GPG PIN) and defers signing to a single bulk step before opening a PR.