spark-submit

name	spark-submit
description	How to submit, monitor, and configure Spark jobs on oleander using the CLI & TypeScript SDK.

oleander Spark Submit

Use this skill when submitting Spark jobs to oleander, monitoring run state, or building automated workflows that execute Spark jobs.

CLI submission

Upload a script and submit it from the terminal:

oleander spark jobs submit my_job.py \
  --namespace my_namespace \
  --name my-job-name \
  --wait

With arguments passed to the script:

oleander spark jobs submit my_job.py \
  --namespace my_namespace \
  --name my-job-name \
  --wait \
  --args "--input-table oleander.default.sf_311 --output-catalog oleander.results"

--wait blocks until the run reaches a terminal state (COMPLETE, FAIL, or ABORT).

With explicit driver and executor sizing:

oleander spark jobs submit my_job.py \
  --namespace my_namespace \
  --name my-job-name \
  --driverMachineType spark.8.c \
  --executorMachineType spark.8.c \
  --executorNumbers 4 \
  --wait

TypeScript SDK submission

Use @oleanderhq/sdk to submit jobs programmatically:

import { Oleander, RunNotFoundError } from "@oleanderhq/sdk";

const oleander = new Oleander(); // reads OLEANDER_API_KEY from env

const { runId } = await oleander.submitSparkJob({
  cluster: "oleander",
  namespace: "my_namespace",
  name: "my-job-name",
  entrypoint: "my_job.py",
});

Submit and wait for completion with built-in polling:

const result = await oleander.submitSparkJobAndWait({
  cluster: "oleander",
  namespace: "my_namespace",
  name: "my-job-name",
  entrypoint: "my_job.py",
});

Polling for run state

When not using submitSparkJobAndWait, poll manually with getRun:

import { Oleander, RunNotFoundError } from "@oleanderhq/sdk";

const oleander = new Oleander();
const { runId } = await oleander.submitSparkJob({ ... });

const started = Date.now();
const timeoutMs = 900_000; // 15 minutes
let notFoundRetries = 10;

while (Date.now() - started < timeoutMs) {
  await wait.for({ seconds: 60 });
  try {
    const run = await oleander.getRun(runId);
    const state = run.state ?? "";
    if (state === "COMPLETE" || state === "FAIL" || state === "ABORT") {
      return { runId, state, run };
    }
  } catch (error) {
    if (error instanceof RunNotFoundError) {
      notFoundRetries--;
      if (notFoundRetries <= 0) throw error;
      continue; // run may not be visible yet; retry
    }
    throw error;
  }
}

throw new Error(`Timeout waiting for run ${runId}`);

Always handle RunNotFoundError — a freshly submitted run may not be immediately visible.

Machine types

Machine types follow the pattern spark.<size>.<class>. Default: spark.1.b for both driver and executors.

Compute (c) — 2:1 RAM:vCPU, CPU-bound transformations:

Type	vCPU	RAM
spark.1.c	1	2 GB
spark.2.c	2	4 GB
spark.4.c	4	8 GB
spark.8.c	8	16 GB
spark.16.c	16	32 GB

Balanced (b) — 4:1 RAM:vCPU, general purpose:

Type	vCPU	RAM
spark.1.b	1	4 GB
spark.2.b	2	8 GB
spark.4.b	4	16 GB
spark.8.b	8	32 GB
spark.16.b	16	64 GB

Memory (m) — 8:1 RAM:vCPU, joins/aggregations that spill:

Type	vCPU	RAM
spark.1.m	1	8 GB
spark.2.m	2	16 GB
spark.4.m	4	30 GB
spark.8.m	8	60 GB
spark.16.m	16	120 GB

Executor count

• Default: 2 executors
• Range: 1–20
• Scale up for large data volumes; keep low for small or exploratory jobs

Testing results after a job run

Ad-hoc SQL query

Run a single SQL query against the lake and print results as a table:

oleander query "SELECT * FROM oleander.my_namespace.my_table LIMIT 20"

Save results as a new table (named by query hash):

oleander query "SELECT id, sum(value) FROM oleander.my_namespace.my_table GROUP BY 1" --save

Interactive DuckDB terminal

Launch a full DuckDB REPL with all registered catalogs pre-attached:

oleander duckdb

Script structure conventions

• Accept table names and output destinations as argparse arguments, not hardcoded.
• Validate required env vars at startup and fail fast with a clear error message.
• Return a non-zero exit code on failure (sys.exit(2)), not just a log message.
• Print a structured JSON summary at the end for observability.