| name | spark-authoring-cli |
| description | Develop Microsoft Fabric Spark/data engineering workflows and write code in Fabric Notebook cells with routing to resources. Provides workspace/lakehouse management, notebook code authoring (PySpark, Scala, SparkR, SQL), and Materialized Lake View (MLV) authoring (Spark SQL MLVs support incremental refresh; PySpark is full-refresh only). Routes to data engineering patterns, development workflow, or infrastructure orchestration. For existing MLV refresh or schedule management, use `mlv-operations-cli`. Triggers: "develop notebook", "Delta Lake patterns", "Spark development", "lakehouse configuration", "write notebook code", "notebookutils", "notebook cell", "PySpark notebook", "%%sql cell", "%%configure", "fabric notebook", "run notebook", "notebook deployment", "materialized lake view", "MLV", "CREATE MATERIALIZED LAKE VIEW", "MLV incremental refresh", "review MLV for incremental refresh", "MLV refresh policy"
|
Telemetry — MANDATORY. Every api.fabric.microsoft.com call must carry
x-ms-fabric-skill: spark-authoring-cli (az rest: --headers "x-ms-fabric-skill=spark-authoring-cli"),
including every LRO poll, fabric_lro and retry. Snippets omit it — add it anyway.
Update Check — ONCE PER SESSION (mandatory)
The first time this skill is used in a session, run the check-updates skill before proceeding.
- GitHub Copilot CLI / VS Code: invoke the
check-updates skill.
- Claude Code / Cowork / Cursor / Windsurf / Codex: compare local vs remote package.json version.
- Skip if the check was already performed earlier in this session.
CRITICAL NOTES
- To find the workspace details (including its ID) from workspace name: list all workspaces and, then, use JMESPath filtering
- To find the item details (including its ID) from workspace ID, item type, and item name: list all items of that type in that workspace and, then, use JMESPath filtering
Spark Authoring — CLI Skill
This skill covers two complementary areas: (1) managing Fabric Spark artifacts via REST APIs (workspaces, lakehouses, notebooks, jobs, pipelines) and (2) writing code inside Fabric Notebook cells (PySpark, Scala, SparkR, SQL with correct lakehouse access, notebookutils, and Spark configuration). For notebook code authoring fundamentals and shared modules, MUST see SPARK-NOTEBOOK-AUTHORING-CORE.md.
Table of Contents
Must/Prefer/Avoid
MUST DO
- Check for recent jobs BEFORE creating new notebook runs — Query job instances from last 5 minutes; if recent job exists, monitor it instead of creating duplicate
- Capture job instance ID immediately after POST — Store job ID before any other operations to enable proper monitoring
- Verify workspace capacity assignment before operations — Workspace must have capacity assigned and active
- When user provides a public data URL, follow the Public URL Data Ingestion policy — keep detailed behavior in the linked resource section to avoid drift/duplication
- Format notebook cells correctly — Each line in cell source array MUST end with
\n to prevent code merging
- Use correct Lakehouse Livy session body format — Send a FLAT JSON with
name, driverMemory, driverCores, executorMemory, executorCores. Do NOT wrap in {"payload": ...} or send only {"kind": "pyspark"} — that causes HTTP 500. Use valid memory values (28g, 56g, 112g, 224g). See Create Lakehouse Livy Session example below and SPARK-CONSUMPTION-CORE.md.
PREFER
- Poll job status with proper intervals — 10-30 seconds between polls; timeout after reasonable duration (e.g., 30 minutes)
- Check job history when POST response is unreadable — If POST returns "No Content" or unreadable response, query recent jobs (last 1 minute) before retrying
- Use Starter Pool for development — Development/testing workloads should use
useStarterPool: true
- Use Workspace Pool for production — Production workloads need consistent performance with
useWorkspacePool: true
- Enable lakehouse schemas during creation — Set
creationPayload.enableSchemas: true for better table organization
- Implement idempotency checks — Prevent duplicate operations by checking existing state first
AVOID
- Never retry POST with same parameters — If you have a job ID, only use GET to check status; don't create duplicate job instances
- Don't skip capacity verification — Operations will fail if workspace capacity is paused or unassigned
- Avoid immediate POST retries on failures — Check for existing/active jobs first to prevent duplicates
- Don't create new runs if monitoring existing job — One job at a time; wait for completion before submitting new runs
- Don't hardcode workspace/lakehouse IDs — Discover dynamically via item listing or catalog search APIs
- Own
%%sql notebook cells here, do not defer to sqldw-consumption-cli — a request to "write a %%sql cell" (or any notebook magic cell) is notebook-cell authoring even when the cell queries a lakehouse table. Only route to sqldw-consumption-cli when the user wants a plain T-SQL query executed against a SQL endpoint, not a notebook cell.
- Own "run/execute the notebook named X" here — running a Fabric notebook by name is notebook execution via the Jobs API (
RunNotebook), which belongs to this skill; do not defer to spark-consumption-cli (that skill is only for ad-hoc Livy session code execution).
- Do NOT use Lakehouse Livy sessions to run a Fabric notebook — Lakehouse Livy sessions (the public Livy API) are for ad-hoc interactive Spark code execution. To run a notebook as a job, use the Jobs API (
RunNotebook) which creates a Notebook Spark session internally. See SPARK-AUTHORING-CORE.md § Notebook Execution & Job Management
- Do NOT schedule MLV refreshes from notebooks — If the user asks to "schedule MLV refresh", route to mlv-operations-cli which uses the REST API. Notebook-based
REFRESH MATERIALIZED LAKE VIEW ... FULL is for one-time manual refresh only, not recurring schedules.
RULES — Read these first, follow them always
Rule 1 — Validate prerequisites before operations.
Verify workspace has capacity assigned (see COMMON-CORE.md Create Workspace and Capacity Management) and resource IDs exist before attempting operations.
Rule 2 — Trust updateDefinition success.
A Succeeded poll result from updateDefinition is sufficient confirmation that content and lakehouse bindings persisted. Do NOT call getDefinition after every upload — it is an async LRO that adds significant latency. Only use getDefinition for its intended purpose: reading current notebook content before making modifications.
Rule 3 — Prevent duplicate jobs and monitor execution properly.
Before submitting new notebook run, ALWAYS check for recent job instances first (last 5 minutes). If recent job exists, monitor it instead of creating duplicate. After submission, capture job instance ID immediately and poll status - never retry POST. See SPARK-AUTHORING-CORE.md Job Monitoring for patterns.
Rule 4 — For notebook code authoring, MUST follow SPARK-NOTEBOOK-AUTHORING-CORE.md.
When writing code inside notebook cells, MUST read SPARK-NOTEBOOK-AUTHORING-CORE.md first — it defines the code generation approach, rules, and a Module Index linking to detailed guides (lakehouse paths, connections, context, orchestration, etc.). Use the Spark-specific resources in this skill (data-engineering-patterns.md, development-workflow.md) for Spark-only implementation details. When the task is about Materialized Lake Views, read materialized-lake-view-patterns.md for authoring/design guidance and mlv-incremental-refresh-patterns.md for refresh-readiness analysis.
Notebook Codegen Quick Rules
Quick reference for common notebook-authoring tasks. The shared common/notebook-authoring/ core (see Rule 4 / SPARK-NOTEBOOK-AUTHORING-CORE.md) is authoritative; if these ever differ, follow the common core.
| User asks for | Required output pattern |
|---|
%%sql / cross-lakehouse query cell | Return a Fabric notebook %%sql cell. Include the named workspace/lakehouse/schema/table in the code or explanatory note. This is notebook authoring, not interactive Spark consumption. |
| Pipeline context detection | Use notebookutils.runtime.context; include isForPipeline = context["isForPipeline"] and read context["currentWorkspaceId"]. |
| Built-in notebook resource files with Spark | Use notebookutils.nbResPath, builtin/, and the file: prefix: spark.read.json(f"file:{notebookutils.nbResPath}/builtin/config.json"). Spark and notebookutils.fs require file: for resource-folder local paths. |
| Fabric connections | Use notebookutils.connections.getCredential("{connectionId}") directly and show a compact code sample. Do not web-search; keep the answer under the tool/turn budget. |
Minimal snippets
context = notebookutils.runtime.context
isForPipeline = context["isForPipeline"]
workspace_id = context["currentWorkspaceId"]
print(f"isForPipeline={isForPipeline}, workspace_id={workspace_id}")
config_df = spark.read.option("multiline", "true").json(
f"file:{notebookutils.nbResPath}/builtin/config.json"
)
display(config_df)
import psycopg2
credential = notebookutils.connections.getCredential("{connectionId}")
conn = psycopg2.connect(
host="<postgres-host>",
database="<database>",
user=credential["username"],
password=credential["password"],
)
Quick Start Examples
For detailed patterns, authentication, and comprehensive API usage, see:
- COMMON-CORE.md — Fabric REST API patterns, authentication, item discovery
- COMMON-CLI.md —
az rest usage, environment detection, token acquisition
- SPARK-AUTHORING-CORE.md — Notebook deployment, lakehouse creation, job execution
Below are minimal quick-start examples. Always reference the COMMON- files for production use.*
Create Workspace & Lakehouse
cat > /tmp/body.json << 'EOF'
{"displayName": "DataEng-Dev"}
EOF
workspace_id=$(az rest --method post --resource "https://api.fabric.microsoft.com" \
--url "https://api.fabric.microsoft.com/v1/workspaces" \
--body @/tmp/body.json --query "id" --output tsv)
cat > /tmp/body.json << 'EOF'
{"displayName": "DevLakehouse", "type": "Lakehouse", "creationPayload": {"enableSchemas": true}}
EOF
lakehouse_id=$(az rest --method post --resource "https://api.fabric.microsoft.com" \
--url "https://api.fabric.microsoft.com/v1/workspaces/$workspace_id/items" \
--body @/tmp/body.json --query "id" --output tsv)
Organize Lakehouse Tables with Schemas
import uuid
def lakehouse_is_schema_enabled():
probe = f"_schema_probe_{uuid.uuid4().hex}"
try:
spark.sql(f"CREATE SCHEMA IF NOT EXISTS {probe}")
spark.sql(f"DROP SCHEMA IF EXISTS {probe}")
return True
except Exception as e:
if "Feature not supported on Apache Spark in Microsoft Fabric" in str(e):
return False
raise
if lakehouse_is_schema_enabled():
spark.sql("CREATE SCHEMA IF NOT EXISTS bronze")
spark.sql("CREATE SCHEMA IF NOT EXISTS silver")
spark.sql("CREATE SCHEMA IF NOT EXISTS gold")
:
()
Create and Refresh a Materialized Lake View (MLV)
CREATE SCHEMA IF NOT EXISTS bronze;
CREATE SCHEMA IF NOT EXISTS silver;
CREATE SCHEMA IF NOT EXISTS gold;
ALTER TABLE bronze.orders_raw SET TBLPROPERTIES (delta.enableChangeDataFeed = true);
CREATE OR REPLACE MATERIALIZED LAKE VIEW silver.orders_clean
(
CONSTRAINT valid_order_id CHECK (order_id IS NOT NULL) ON MISMATCH DROP
)
AS
SELECT
order_id,
customer_id,
CAST(order_ts AS TIMESTAMP) AS order_ts,
amount
FROM bronze.orders_raw;
REFRESH MATERIALIZED LAKE silver.orders_clean ;
Create Lakehouse Livy Session
cat > /tmp/body.json << 'EOF'
{"name": "dev-session", "driverMemory": "56g", "driverCores": 8, "executorMemory": "56g", "executorCores": 8, "conf": {"spark.dynamicAllocation.enabled": "true", "spark.fabric.pool.name": "Starter Pool"}}
EOF
az rest --method post --resource "https://api.fabric.microsoft.com" \
--url "https://api.fabric.microsoft.com/v1/workspaces/$workspace_id/lakehouses/$lakehouse_id/livyapi/versions/2023-12-01/sessions" \
--body @/tmp/body.json
Lakehouse Livy Session Body — Common Mistakes
- ❌
{"payload": {"kind": "pyspark"}} → HTTP 500 (wrong wrapper, missing required fields)
- ❌
{"kind": "pyspark"} → HTTP 500 (missing driverMemory, executorMemory, etc.)
- ✅ Flat JSON with
name, driverMemory, driverCores, executorMemory, executorCores (and optionally conf with Starter Pool)
Spark Performance Configs
For detailed workload-specific configurations, see data-engineering-patterns.md Delta Lake Best Practices.
Quick reference:
Focus: Essential CLI patterns for Spark/data engineering development and notebook code authoring, with intelligent routing to specialized resources. For comprehensive patterns, always reference COMMON-* files and resource documents.
