Migrate Spark scripts and notebooks to Snowflake. Routes to one of two bundled conversion paths and orchestrates the post-conversion pipeline. **Default path: Snowpark Connect (SCOS)**, which preserves the PySpark API surface. The SMA / Snowpark API path is invoked only when the user explicitly asks for it. Also **deploys** the output as a **Notebook** (`deploy-notebook`) or **Code Bundle** (`deploy-code-bundle`).
Triggers: convert spark, migrate pyspark, migrate spark, migrate to snowpark, convert to snowpark, snowpark connect, scos, scos migration, migrate to snowpark connect, migrate to scos, snowpark api, sma cli, sma conversion, run sma, snowflake.snowpark rewrite, already migrated, already ran sma, sma dashboard, fix ewis, stage conversion, dvp orchestrator, resume dvp, assess pyspark, assess spark, assess databricks, spark assessment, databricks assessment, spark workload assessment, migration readiness, migration readiness report, spark readiness, spark compatibility, spark compatibility report, pyspa
Standardmäßig ist der Prompt ausgewählt, der zuerst die Quelle prüft. Sie können zu einem direkten Befehl wechseln oder eine lokale Kopie herunterladen.
Quelldateien prüfen
Lesen Sie SKILL.md und alle von SkillsMP angezeigten Begleitdateien, bevor Sie sich für eine Installation entscheiden.
Mit Codex oder Claude installieren Kopieren Sie diesen Prompt, fügen Sie ihn in Codex, Claude oder einen anderen Assistant ein und lassen Sie die Skill-Seite prüfen und installieren.
Ein direkter Befehl überspringt den Prüf-Prompt. Prüfen Sie die Quelle, bevor Sie ihn ausführen.
Migrate Spark scripts and notebooks to Snowflake. Routes to one of two bundled conversion paths and orchestrates the post-conversion pipeline. **Default path: Snowpark Connect (SCOS)**, which preserves the PySpark API surface. The SMA / Snowpark API path is invoked only when the user explicitly asks for it. Also **deploys** the output as a **Notebook** (`deploy-notebook`) or **Code Bundle** (`deploy-code-bundle`).
Triggers: convert spark, migrate pyspark, migrate spark, migrate to snowpark, convert to snowpark, snowpark connect, scos, scos migration, migrate to snowpark connect, migrate to scos, snowpark api, sma cli, sma conversion, run sma, snowflake.snowpark rewrite, already migrated, already ran sma, sma dashboard, fix ewis, stage conversion, dvp orchestrator, resume dvp, assess pyspark, assess spark, assess databricks, spark assessment, databricks assessment, spark workload assessment, migration readiness, migration readiness report, spark readiness, spark compatibility, spark compatibility report, pyspark compatibility, pyspark compatibility report, databricks compatibility, databricks to snowflake, pyspark to snowflake, spark to snowflake, spark snowflake, migration feasibility, migration effort, migration effort estimate, migration complexity, migration scope, how hard is migration, how complex is migration, analyze spark, analyze pyspark, analyze databricks, spark analysis, pyspark analysis, workload analysis, pre-migration analysis, spark workload analysis, check spark, check pyspark, check databricks, check compatibility, spark compatibility check, scan spark, scan pyspark, spark audit, audit spark, review spark, understand spark, understand pyspark, understand databricks, evaluate spark workload, should I migrate to snowflake, can my spark run on snowflake, snowflake readiness, scope the migration, deploy notebook, create notebook from migration output, deploy code bundle, create code bundle from migration output.
Spark Migration
Routes Spark → Snowflake conversion requests to one of two bundled paths and owns the cross-path configuration, intent classification, and per-step dispatch.
⛔ Default Path: Snowpark Connect (SCOS)
Any unqualified request to "convert spark", "migrate pyspark", "migrate to snowpark", etc. routes to , which preserves the PySpark API surface (, , ). This is the recommended default for the vast majority of users.
snowpark-connect/SKILL.md
withColumn
groupBy
spark.createDataFrame
Route to snowpark-api/SKILL.md (SMA CLI / snowflake.snowpark snake_case rewrite) only when the user explicitly mentions:
"SMA", "SMA CLI", or "run sma"
"Snowpark API" or "snowflake.snowpark"
An already-migrated SMA output (intents like "open sma dashboard", "fix ewis", "run stage conversion", "resume dvp")
An existing project config with conversion_type set to snowpark-api (or any legacy alias)
If the intent is ambiguous, ask the user which path they want — do not silently default to the SMA path.
⛔ Deploy Path (notebook or code bundle)
When the user wants to deploy the migration's output to Snowflake (rather than convert Spark), route to the deploy sub-skill matching the requested target:
Notebook → deploy-notebook/SKILL.md — uploads the project (.ipynb main file + supporting files) to a stage and creates a Notebook. Intents: "deploy notebook", "deploy the migrated notebook", "create notebook from migration output".
Code Bundle → deploy-code-bundle/SKILL.md — composes an entrypoint, writes code_bundle.yml, uploads the .py project to a stage, and creates a Code Bundle that runs on a warehouse or compute pool. Intents: "deploy code bundle", "run as code bundle", "execute code bundle", "create code bundle from migration output".
See Step 6: Deploy Dispatch.
Flows
(a) Already migrated — User already has SMA/snowpark-connect output. Provide the result path, verify structure, initialize git, then run the post-conversion pipeline.
(b) Snowpark Connect conversion — Load snowpark-connect/SKILL.md. This is the default.
(d) Deploy — Deploy the migration's output to Snowflake as a Notebook (deploy-notebook/SKILL.md) or a Code Bundle (deploy-code-bundle/SKILL.md): upload the migrated project to a stage and create the object. See Step 6: Deploy Dispatch.
Output Format
Every time you begin a step, sub-step, or significant action, prefix the message with a timestamp in the format [YYYY-MM-DD HH:MM:SS]. Obtain the current time by running date '+%Y-%m-%d %H:%M:%S' in bash.
⛔ Final Summary: the active conversion path owns its own Final Summary template. For the Snowpark API path, see snowpark-api/references/final-summary-template.md. For the Snowpark Connect path, follow snowpark-connect/SKILL.md. Do NOT improvise your own summary format.
Sub-skill Loading Convention
All sub-skills referenced by name in this document (snowpark-connect, snowpark-api, deploy-notebook, deploy-code-bundle, and snowflake-notebook-migration) are bundled inside this skill's own directory tree — they are NOT separately installed top-level skills and they will NOT appear in the skill registry. This is intentional: the parent owns user-facing triggers; sub-skills are workers loaded on demand to avoid trigger collisions.
To run a sub-skill (NOT via skill("<name>")):
Resolve its SKILL.md path relative to this SKILL.md's directory (<skill_directory>):
Read that file with the Read tool and follow its instructions verbatim, passing the orchestrator context inline as the next turn's content.
If the file is missing at the expected path, STOP and report:
The bundled <name> sub-skill is missing at <path>. Reinstall the spark-migration skill.
Do NOT fall back to a registry lookup — that lookup will fail.
The snowpark-api/ sub-skill further loads its own children (migrate-pyspark-to-snowpark-api, validate-pyspark-to-snowpark-api, sma-dashboard-generator, stage-conversion, dvp/dvp-*) using the same convention — see snowpark-api/SKILL.md for its own loading table.
Not found: ask the user for the absolute path to the sma binary (typically /Users/me/SMA-CLI-arm64-mac/orchestrator/sma). Validate with test -x "<path>". On valid path, save globally as above. Do not proceed without a verified SMA CLI.
The live progress dashboard is an experimental Snowpark Connect (SCOS)
feature and is OFF by default. Do not launch it here — at this point
the conversion path is not yet resolved, and the SMA / snowpark-api path does
not use this UI.
Only launch the UI when the user explicitly asks for it (e.g. "show the
progress UI", "open the migration dashboard"). If the user has not explicitly
asked, the dashboard must never start — there must be no accidental launches.
Analysis-only flows skip this step entirely. If the intent is assessment /
readiness report (routing to assess-pyspark-workload, or loading a Scala/Java
migration skill to stop after Phase 1a), do not mention, offer, or launch the
Progress UI — it only makes sense for full multi-phase migrations.
When (and only when) the user has explicitly requested it and the flow is a
full migration, the Snowpark Connect path launches the server itself, once the
conversion folder (<CONVERSION>) is resolved, so the server tails the exact
event log the SCOS phases write to. See the Progress UI Emit Hooks
(Experimental) section of
snowpark-connect/migrate-pyspark-to-snowpark-connect/SKILL.md.
No action is required in this step. The launcher enforces this with a
--confirm-experimental interlock (a no-op without it), honours
<config.enable_progress_ui> (default no), is idempotent, and is non-fatal —
migration never blocks on the UI.
Step 1: Load Configuration
Record <start_time> = current time (used for duration in the Final Summary).
load merges defaults for missing keys AND normalizes legacy conversion_type aliases (scos, snowpark_connect, snowpark_api) to their canonical forms (snowpark-connect, snowpark-api), persisting the normalized form back to disk.
Store the result as <config> and <config_path> for later writes. Go to Step 2.
1.3 Create New
Ask for the 5 required project fields in one numbered list:
New configuration — please provide the following:
1. Project Name: (used as configuration filename)
2. Source Code Path: (PySpark or Spark Scala source directory)
3. Output Folder: (where converted code will be saved)
4. Customer Email:
5. Customer Company:
Example: "1. my_project, 2. /Users/me/spark-etl, 3. /Users/me/output, 4. user@co.com, 5. Acme Inc"
#1 (Project Name) is required. #2–#5 may be deferred.
When <config.migration_status> = already_migrated:
Ask for the output path (pre-fill <config.output_folder>). Store as <output>.
test -d "<output>/Output" && test -d "<output>/Reports" && echo"Valid" || echo"Invalid"
If invalid, check for SMA v1 Conversion-* subfolder:
ls -d "<output>"/Conversion-* 2>/dev/null | sort | tail -1
If present, resolve <output> to it and re-validate. If still invalid, ask the user for the correct path. (Full v1/v2/v3 resolution rules: snowpark-api/references/output-layouts.md.)
Read with the Read tool and follow inline. Context block:
Parameter
Value
Sub-skill variable
Source path
<input>
$ARGUMENTS
Output path
<output>
$OUTPUT
Customer Email
<email>
$EMAIL
Customer Company
<company>
$COMPANY
Project Name
<project>
$PROJECT
Invoker identity
orchestrator
snowpark_connect_invoker
Migration status
<config.migration_status>
Pass through unchanged
Include snowpark_connect_invoker: orchestrator verbatim — the SCOS sub-skills read this flag to suppress their standalone Phase-6 notebook handoff (which would otherwise duplicate the work owned by snowflake-notebook-migration).
The SCOS sub-skill owns the entire conversion AND post-conversion pipeline for its path; control does not return here.
Read with the Read tool and follow inline. Context block:
The following context was configured by the spark-migration orchestrator:
- <intent> = migrate | already_migrated (per Step 3 routing)
- <input> = <config.input_folder>
- <output> = <config.output_folder> (already validated for already_migrated)
- <email> = <config.email>
- <company> = <config.company>
- <project> = <config.project_name>
- <config_path> = <skill_directory>/configurations/<project>.json
- <spark_migration_root> = <skill_directory>
- <snowpark_api_root> = <skill_directory>/snowpark-api
- <start_time> = <start_time>
Detect source language and route to the appropriate child.
Do NOT return here — the API path owns its own final summary.
The snowpark-api/ router will:
Detect language (Python only is supported; Scala stops with a switch-to-SCOS prompt)
Route by <intent> to either migrate-pyspark-to-snowpark-api/ (fresh conversion) or validate-pyspark-to-snowpark-api/ (already-migrated / individual operations)
The chosen child runs the entire pipeline through the Final Summary
Step 6: Deploy Dispatch (Flow d — deploy intent)
When the request is to deploy the migration's output (not to convert Spark), skip Steps 3–5 and dispatch to the deploy sub-skill for the requested target.
Notebook target — Readdeploy-notebook/SKILL.md with the Read tool and follow it inline:
The deploy-notebook/ sub-skill runs its own steps (remind-to-fix → verify namespace → upload the project to a stage → create the Notebook) and owns its Final Summary. Control does not return here.
Code Bundle target — Readdeploy-code-bundle/SKILL.md with the Read tool and follow it inline:
the file to run (generated in Step 0.6), relative to the project dir
$ENTRYPOINT
Target db/schema/stage
as configured
$DATABASE/$SCHEMA/$STAGE
Bundle name
code bundle object name
$BUNDLE_NAME
Compute
warehouse (default) or compute_pool
$COMPUTE_TYPE/$COMPUTE_POOL/$WAREHOUSE
Connection
Snowflake connection name
$CONNECTION
The deploy-code-bundle/ sub-skill runs its own steps (remind-to-fix → verify namespace → compose entrypoint → prepare spec → upload → create the bundle) and owns its Final Summary. Control does not return here.
Direct-Intent Triggers
When the user opens a session with intents like "fix ewis", "open sma dashboard", "run stage conversion", or "resume dvp", route them as follows:
Run Step 0 (prerequisites). SMA CLI check is skipped — these intents operate on existing output.
Run Step 1 to locate or create a config. Force-set conversion_type=snowpark-api and migration_status=already_migrated if not already.
Run Step 4 to validate <output>.
Skip Step 5 dispatch and load snowpark-api/validate-pyspark-to-snowpark-api/SKILL.md directly, passing the intent (fix ewis / open sma dashboard / etc.) in the context block — the validator's "Direct-Intent Behavior" table maps each intent to the subset of steps to run.
For deploy intents ("deploy notebook", "create notebook from migration output", "deploy code bundle", "run as code bundle", "create code bundle from migration output"), go directly to Step 6: Deploy Dispatch — do not run the conversion Steps 3–5.
SMA CLI cannot be located/validated (API path only)
Stop in Step 0.2; ask user for path
<config.conversion_type> is snowpark-api but user clearly wants SCOS
Ask the user to confirm; offer to flip via Edit settings
<output> exists but lacks Output/ and Reports/ (already_migrated)
Stop in Step 4; ask for correct path
Bundled sub-skill missing at expected path
Stop and report: "The bundled <name> sub-skill is missing at <path>. Reinstall the spark-migration skill."
Per-step error handling lives inside the bundled sub-skills. See snowpark-connect/SKILL.md and snowpark-api/SKILL.md.
Outputs
Both conversion paths produce a converted workload with:
Output
Location
Converted code
<output>/Output/
Issues report
<output>/Reports/Issues.csv
Inventory
<output>/Reports/InputFilesInventory.csv
Dependency inventory
<output>/Reports/ArtifactDependencyInventory.csv
EWI dashboard (snowpark-api)
<output>/sma-dashboard/
DVP workspace (snowpark-api)
<output>/dvp/
The Snowpark Connect path produces an equivalent but path-distinct Reports/ layout — see snowpark-connect/SKILL.md.
Example Workflows
See snowpark-api/references/example-workflows.md for concrete conversational walkthroughs of the four common entry points (fresh SMA, already-migrated, direct intent, Scala fallback).
For Snowpark Connect example walkthroughs, see snowpark-connect/SKILL.md.
