Menu
oleander-specific Spark guidance for connected OpenLineage, collect() pitfalls, and environment variable usage.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Based on SOC occupation classification
| name | spark-lineage |
| description | oleander-specific Spark guidance for connected OpenLineage, collect() pitfalls, and environment variable usage. |
Use this skill when Spark lineage in oleander looks disconnected or when rewriting Spark jobs to preserve connected lineage.
Key behavior from oleander Spark/OpenLineage integrations:
collect() is a Spark action that materializes data into driver memory.collect(), execution is regular Python in-memory logic, not distributed Spark DataFrame execution.Prefer this shape for connected lineage:
df = spark.table(...)select, withColumn, join, aggregate)df.write(...)If collect() is required, keep it for small side-effects or reporting, not as the core bridge between read and write.
When fixing lineage gaps:
collect, toPandas, driver loops).df.write...), not Python-memory writes.Use env vars for runtime configuration, not transformation logic.
os.getenv("VAR", "default")Example pattern:
import os
job_name = os.getenv("NAME", "default-service")
output_catalog = os.getenv("OUTPUT_CATALOG", "oleander.sf")
In this repo, examples/print_name.py uses:
NAME with fallback default-serviceUse lake-query to explore your data; compare seasonal revenue, analyze product changes, count new user signups, or find whatever answers are hidden in your data.
How to submit, monitor, and configure Spark jobs on oleander using MCP, the CLI, and the TypeScript SDK.
Run Polars queries or scripts on oleander, locally or distributed, and save results to the lake catalog.
Engine-agnostic guidance for working with the oleander lake catalog, including naming conventions, catalog hierarchy, and table reference patterns.
Spark-specific patterns for reading and writing oleander lake catalog tables, including append vs overwrite, avoiding driver writes, and reusable table naming.
General Apache Spark best practices for scalable, maintainable, and performant DataFrame jobs.