fabric-spark

# Spark / PySpark in Fabric ## Key Constraints - Fabric Spark cannot access arbitrary external HTTP/HTTPS URLs — land data in lakehouse `Files/` first (via pipeline Copy activity, OneLake API, or curl) - Use `abfss://` URI format for OneLake paths in Spark: `abfss://{workspace}@onelake.dfs.fabric.microsoft.com/{item}.Lakehouse/{path}` - Use workspace GUIDs (not names) in ABFS URIs — spaces are not allowed - `mssparkutils` for Fabric-specific notebook operations (credentials, secrets, file management) - Use Delta Lake format for all Lakehouse tables ## Runtime Context vs Spark Session Config Two different things that are often confused: | Need | API | |---|---| | Workspace / item identity (workspace ID + name, notebook ID + name, default lakehouse ID + name, userId) | `notebookutils.runtime.context["currentWorkspaceId"]` (etc.) — a dict, documented public API, works in pure-Python notebooks | | Spark session tuning (shuffle partitions, AQE, Delta settings, case sensitivity) | `spark.conf.set(...)` / `spark.conf.get(...)` | `spark.conf.get("trident.workspace.id")` also returns the workspace ID but is internal Spark conf, not documented surface, and is unavailable in pure-Python notebooks. Prefer `notebookutils.runtime.context` for identity lookups; reserve `spark.conf.*` for session tuning. ## Lakehouse Setup - **`enableSchemas` is set at lakehouse creation time only** — cannot be retrofitted. Without it the lakehouse only has the default `dbo` schema and you must recreate to gain named schemas. Set via `creationPayload: { "enableSchemas": true }` on `POST /workspaces/{ws}/items` (see fabric-rest-api skill). - Schemas use lowercase names by convention (`bronze`/`silver`/`gold` for medallion). `DROP SCHEMA <name> CASCADE` removes the schema with all its tables. - **Cross-lakehouse Spark SQL** uses 3-part names: `lakehouse.schema.table` for same-workspace, `workspace.lakehouse.schema.table` for cross-workspace. Verify access permissions on each lakehouse. - Lakehouse delete cascades irreversibly: SQL Endpoint deleted, all OneLake data permanently removed, shortcuts pointing in become inaccessible, dependent notebooks fail at runtime. - **Shortcuts as definition payload**: when authoring a Lakehouse via REST, `shortcuts.metadata.json` is an array of `{name, path, target}` objects. Supported `target.type` values: `OneLake`, `AdlsGen2`, `AmazonS3`, `GoogleCloudStorage`, `S3Compatible`, `Dataverse`. Each target type has its own connection properties (see fabric-rest-api skill). ## Lakehouse Table Maintenance (impacts SQL Endpoint performance) - Run `OPTIMIZE` regularly to compact small files (target 128 MB – 1 GB per file) - Run `VACUUM` to remove unreferenced files - V-Order write optimization is default — do not disable - Avoid high-cardinality partition columns; aim for partitions ≥ 1 GB - SQL Endpoint metadata sync lag is normally < 1 minute but increases with: many lakehouses per workspace, small-file fragmentation, large ETL volume, or SQLEP idle > 15 min (this describes the legacy background sync) ### New SQL analytics endpoint metadata sync (PREVIEW, May 2026) Opt-in faster sync that keeps data queryable within seconds of landing. **Preview** — and applies to **NEW SQL analytics endpoints only**: existing endpoints in the workspace stay on the legacy sync above. - **Enable**: Workspace settings → Warehouse → New metadata sync (preview). Only endpoints created *after* enabling get the new sync. - **Architecture**: external-tables-based Delta-log parsing with decoupled schema-vs-data change detection (schema changes and data changes refresh separately), plus a periodic background refresh and on-demand refresh when a read query hits stale data. - **New DMV** `sys.dm_db_external_tables_log_status` — `last_update_time_utc`, `latest_log_version`, `latest_checkpoint_version`, `is_blocked` (`1` = last update blocked, `0` = succeeded). - **Targeted manual refresh** (new-sync endpoints only, for data-only changes): ```sql EXEC sys.sp_dw_refresh_ext_table 'dbo.<table>'; ``` For schema changes (add/drop tables or columns, type changes) use the full-item Refresh SQL endpoint metadata REST API instead. - **Limitations**: no support for multi-part checkpoint (a deprecated Delta feature — tables containing them fail to update); cannot be enabled when the workspace uses workspace private link. ## Notebook REST API / UI Upload When creating notebooks via REST API, every code cell must include `"outputs": []` and `"execution_count": null`. **Cell `source` must be an array of strings, not a bare string.** nbformat permits either, but Fabric's UI upload (`createArtifact`) and the REST definition APIs reject the bare-string form with a generic `400 exceptionCulprit:1` that gives no clue which field is wrong. Split on `\n` and append `\n` to every line except the last — the standard nbformat convention: ```json "source": ["line one\n", "line two\n", "last line"] ``` Not: ```json "source": "line one\nline two\nlast line" ``` Applies to every cell in the notebook, markdown and code. A single bare-string source anywhere in the file fails the whole upload. **Default lakehouse binding** uses `metadata.dependencies.lakehouse` in the notebook content — auto-mounts at runtime so `spark.read.table("schema.table")` resolves without 3-part naming: ```json { "metadata": { "dependencies": { "lakehouse": { "default_lakehouse": "<lakehouse-id>", "default_lakehouse_workspace_id": "<workspace-id>", "default_lakehouse_name": "<lakehouse-name>" } } } } ``` One default lakehouse per notebook. Additional lakehouses are reachable via 3-part names (see Lakehouse Setup). ### Definition API gotchas - `getDefinition` is a POST, not GET — empty body returns **HTTP 411 Length Required**. Always send `'{}'` as the body. - After 202 + `Location` header, poll `GET {Location}` until `Succeeded`, then call **`GET {Location}/result`** (note the `/result` suffix) to retrieve the actual content. Without `/result` the operation reports `Succeeded` but returns no payload. - `updateDefinition?updateMetadata=true` requires a `.platform` part in `definition.parts`; the flag without `.platform` returns 400. For content-only updates omit the flag entirely. - Conversely, **omitting `?updateMetadata=true` silently ignores any `.platform` part** in your payload — `displayName`/`description` won't update. ## Notebook execution via REST `POST /v1/workspaces/{ws}/items/{itemId}/jobs/instances?jobType=RunNotebook` (see fabric-rest-api skill for the full jobType table). - **`defaultLakehouse` requires both `id` AND `name`** in the execution config. Supplying only `id` returns 400 — common cause of "DefaultLakehouse: missing name" errors. - **Pool selection** via `executionData.configuration`: `useStarterPool: true` (dev/shared), `useWorkspacePool: true` (prod), or a custom pool name (high-memory/GPU). Starter pool falls back when the workspace pool is at capacity. - **Job states**: `NotStarted → Running → Completed | Failed | Cancelled`. Poll `GET {Location}` from the 202 response, or `GET .../jobs/instances/{jobInstanceId}` if you captured the ID. - **Never retry POST after a network/timeout error.** Query `GET .../jobs/instances` filtered to the last 5 minutes first; if a recent run exists, monitor that. Retrying creates duplicate runs and burns CUs. - Job stuck in `NotStarted` longer than ~2 minutes usually means pool warm-up or capacity SKU contention, not a notebook bug. ### In-notebook auto-restart (`%%configure retriableOptions`) For pipeline-driven notebook runs, the notebook itself can opt into automatic restart after system failures using `%%configure` (April 2026): ```python %%configure { "retriableOptions": { "enabled": true, "maxAttempt": 3 } } ``` `enabled` is the on/off switch; `maxAttempt` (singular) caps total attempts. Place it as the first cell, same as any other `%%configure`. This complements — does **not** replace — the pipeline activity-level retry; configure one layer or the other to avoid stacking retries that multiply CU consumption. Distinct from the "never retry POST" rule above: that rule applies to **external orchestrators** submitting jobs via REST; `retriableOptions` is evaluated **inside** Fabric's notebook runtime. New feature — verify the schema against current Microsoft Learn before relying on it in production. ## Other Spark item definitions ### SparkJobDefinition | Format | Required parts | |---|---| | `SparkJobDefinitionV1` | `SparkJobDefinitionV1.json` | | `SparkJobDefinitionV2` | `SparkJobDefinitionV1.json` (yes — the file name still says V1 in V2 format), plus `Main/<file>` and optional `Libs/<file>` | V2 only accepts `.py` and `.R` files in `Main/` and `Libs/` — **JAR files are not supported in V2 parts**. The JSON schema includes `executableFile`, `defaultLakehouseArtifactId`, `mainClass`, `additionalLakehouseIds`, `commandLineArguments`, `additionalLibraryUris`, `language`, `environmentArtifactId`. ### Environment Default format (omit `format` or set to `null`). Key parts: | Part Path | Content | |---|---| | `Libraries/PublicLibraries/environment.yml` | Conda / pip dependencies | | `Setting/Sparkcompute.yml` | Pool config: `driver_cores`, `driver_memory`, `executor_cores`, `executor_memory`, `dynamic_executor_allocation` (`min_executors` / `max_executors`), `runtime_version` | | `Libraries/CustomLibraries/<name>.{jar\|py\|whl\|tar.gz}` | Custom user uploads — JAR + Python + wheel + R archive all supported here (unlike SparkJobDefinition V2 `Main`/`Libs`) | ## Reference - Microsoft Learn: [What is a lakehouse in Microsoft Fabric?](https://learn.microsoft.com/fabric/data-engineering/lakehouse-overview) - Microsoft Learn: [Apache Spark compute in Microsoft Fabric](https://learn.microsoft.com/fabric/data-engineering/spark-compute) - Microsoft Learn: [NotebookUtils (formerly MSSparkUtils) for Fabric](https://learn.microsoft.com/fabric/data-engineering/notebook-utilities) - Comprehensive MS Learn link bundle (concept / notebooks / lakehouse / performance / SJD / environments / runtime / best practices): [references/REFERENCE.md](references/REFERENCE.md) ## See also - fabric-rest-api skill — notebook definition upload API and LRO pattern - fabric-error-handling skill — Tier 1/2 convention for notebook code - fabric-warehouse-monitoring skill — Query Insights for SQLEP queries against lakehouse tables - fabric-spark-monitoring skill — Livy session listings, Spark History Server mirror, notebook wall-clock attribution