| name | aidp-migrator-overview |
| description | Router skill. Read this first whenever the user mentions migrating a Databricks workload (notebooks, jobs, catalogs, schedules) onto Oracle AI Data Platform (AIDP). Lays out the toolkit, the two-pass migration architecture, and which of the other aidp-* skills handles each phase. Compose those skills; this one adds no API surface. |
aidp-migrator-overview — router
Migrating a Databricks workload onto AIDP is a multi-phase operation. This skill picks the right next skill based on what the user asks.
When to use
- The user mentions "migrate Databricks", "port from Databricks to AIDP", "Unity Catalog migration", "DBX → AIDP", "lift-and-shift Databricks job", or similar.
- The user asks "where do I start" with the migrator toolkit.
- The user asks for an overview of the migrator architecture.
The two-pass architecture (mental model)
┌────────────────────┐ ┌──────────────────────┐ ┌─────────────────────┐
│ Pass-0: Plan │ → │ Pass-1: Dep code │ → │ Pass-2: Execute │
│ build_dag.py │ │ ensure_migrated() │ │ job_migrate.py │
│ check_data_ │ │ walks %run tree, │ │ cell-by-cell on a │
│ availability.py │ │ rewrites Databricks │ │ live AIDP cluster, │
│ │ │ APIs in each dep │ │ 4-way verify, up │
│ (read-only) │ │ notebook (no run) │ │ to 10 fix attempts │
└────────────────────┘ └──────────────────────┘ └─────────────────────┘
The catalog migration is a SEPARATE flow, run BEFORE Pass-2 (so the
schemas + table locations exist when migrated notebooks try to read them):
┌──────────────────────────────────┐ ┌──────────────────────────────────┐
│ extract_catalog_databricks.py │ → │ migrate_catalog.py │
│ REST against Unity Catalog API │ │ 18 DDL rewrite rules → batched │
│ → reports/catalog_pack.json │ │ CREATE SCHEMA / CREATE TABLE │
│ │ │ on AIDP in a single WS execute │
└──────────────────────────────────┘ └──────────────────────────────────┘
Pick the right skill for the user's ask
| User says | Skill to invoke |
|---|
| "Migrate this Databricks job", "port this workflow", "convert to AIDP" | aidp-migrate-job |
| "Build a migration manifest", "what would migrate", "show the DAG" | aidp-build-dag |
| "Are my source tables available?", "pre-migration check", "is the data ready" | aidp-check-data |
| "Resume the migration", "skip already-migrated", "pick up where I left off" | aidp-resume-migration |
| "Cell N is failing", "fix this notebook", "retry from cell K" | aidp-fixup-cell |
| "Migrate the Unity Catalog", "port the HMS schemas", "DDL migration" | aidp-migrate-catalog |
| "Map s3 buckets to OCI", "configure bucket mapping" | aidp-bucket-mapping |
| "Streaming convergence", "acceptance contract", "wait for pipeline to settle" | aidp-acceptance-contract |
| First time using this toolkit, "what do I need to install" | aidp-migrator-bootstrap |
What the user must have set up before any of this
This plugin is self-contained for Codex. The OpenAI-based migrator engine is bundled under engine/ and staged by the SessionStart hook to ~/.aidp-migrator/engine. Before any skill in this plugin can do real work, the user needs:
- Bundled engine staged locally. Start a new Codex thread after install, or run
python <plugin-root>/hooks/session_start.py once. Skill aidp-migrator-bootstrap verifies this.
~/.oci/config with either an api_key profile (unattended) or session-token profile (interactive).- An ACTIVE AIDP cluster. The migrator's Pass-2 requires a live cluster — the WebSocket execute path. If the cluster is stopped, ask the user to start it via AIDP console before invoking
aidp-migrate-job.
OPENAI_API_KEY in the environment. The migrator uses OpenAI model with tool use for every cell rewrite. Without this key the Pass-2 loop won't run.- An
env-coords.md file — see references/env-coords.md. The customer fills in their DataLake OCID, workspace UUID, cluster ID, AIDP base URL, OCI profile name once; every other skill threads these through.
What this plugin does NOT do
- It does not migrate Databricks
dbutils.fs to OCI Object Storage files — only table/notebook/job constructs. File-level DBFS replication is a separate exercise.
- It does not handle Databricks Workflows-on-Pipelines (DLT) — only Jobs + tasks. DLT pipelines need manual recreation as Spark structured streaming + scheduling.
- It does not migrate Databricks ML feature-store registrations or MLflow model versions automatically. Those need separate handling.
- It does not provision the AIDP cluster, workspace, or DataLake. Assume those exist.
Key references
Order of operations for a fresh migration
aidp-migrator-bootstrap — once per workstation.aidp-migrate-catalog — schemas and tables FIRST, so notebook reads have targets.aidp-bucket-mapping — only if migrating tables with s3:// external locations.aidp-build-dag — produces reports/<job>_manifest.json.aidp-check-data — verify before committing cluster time.aidp-migrate-job — the big run.aidp-fixup-cell (only if needed) for cells the auto-fix loop couldn't recover.aidp-acceptance-contract (for streaming / batch convergence pipelines).
