flowx-setup

# Set up flowx flowx runs its phases (`discover`, `convert`, `package`, `migrate`) in one of two ways. This skill prepares whichever fits your environment, keyed on `DATABRICKS_RUNTIME_VERSION` (the same signal the rest of the plugin uses to detect Databricks): - **Databricks Genie Code** (`DATABRICKS_RUNTIME_VERSION` *set*) → the phases run as **MCP tools** hosted on a Databricks App. **No virtual environment is created** — the app vendors its own copy of the flowx code and dependencies, so the phase skills just call the single `flowx` MCP tool. - **Local / Claude Code / other agents** (`DATABRICKS_RUNTIME_VERSION` *unset*) → the phases run from a **Python virtual environment** via the CLI, with an optional local (stdio) MCP server. ## Step 1 — Pick the path ```bash if [ -n "${DATABRICKS_RUNTIME_VERSION:-}" ]; then echo "Databricks / Genie Code → deploy the MCP server (Path A, no venv)" else echo "Local → create the virtual environment (Path B)" fi ``` ## Step 2 — Ask about the live Azure features Ask the user before installing anything: > Do you want the live Azure features: pull ADF definitions with `extract` and estimate current > spend with `profile`? They need extra Azure packages (`azure-identity`, `requests`) and Azure > credentials. Remember the answer. On **yes**, the steps below add flowx's optional `azure` extra; on **no**, they install the base dependencies only (the default). You can re-run setup later to add it. --- ## Path A — Databricks Genie Code (MCP, no virtual environment) In Genie Code the phases run on the deployed app, so **do not run `bootstrap.sh` and do not create a venv** — it isn't needed. Deploy the MCP server instead. **Recommended (works on serverless): run the `app/deploy_app.py` notebook.** It uses the Databricks SDK to stage a self-contained bundle (the app entrypoint plus a vendored copy of the flowx source) to **`/Workspace/Shared/mcp-flowx`** and create/deploy the **`mcp-flowx`** Databricks App. Set its `repo_root` widget to the flowx checkout (e.g. `/Workspace/Shared/flowx`). Because it uploads through the SDK Workspace API, it runs directly in a serverless Genie Code session. Set its `source_deps` widget to `azure` if the user said yes in Step 2 (it adds the Azure deps to the app); otherwise leave it blank. The notebook prints the app URL; the MCP endpoint is `<app-url>/mcp`. **CLI alternative:** `bash <plugin_dir>/app/deploy.sh` does the same via the Databricks CLI (`apps deploy` / `sync`); prefix it with `SOURCE_DEPS=azure` if the user said yes in Step 2. Those commands require a CLI session — run it from a workspace web terminal or a local machine, **not** serverless notebook Python. > **Clone into a shared location.** The app's service principal cannot read private > `/Workspace/Users/<you>` folders by default, so `deploy.sh` deploys the source from > `/Workspace/Shared/<app-name>`. Clone flowx into a Git folder under **`/Workspace/Shared`** > (e.g. `/Workspace/Shared/flowx`), not your user home. If `/Workspace/Shared` is restricted, > use another all-users location and pass it via `APP_SOURCE_PATH`. After it deploys, relay these follow-up steps to the user (the script also prints them): 1. **App access:** grant **Can use** on `mcp-flowx` to the users / service principals that will call it (Apps UI → *Permissions*, or `databricks apps set-permissions mcp-flowx ...`). 2. **Data access:** grant the app's service principal access to the catalogs, schemas, and volumes the migration touches (plus any SQL warehouse used by the reporting tools). 3. **Add it in Genie Code (Agent mode):** open Genie Code **Settings → MCP Servers → Add Server**, choose **Custom MCP server**, select the `mcp-flowx` app, and **Save**. The single `flowx` tool then appears (MCP needs Agent mode; it uses one of the 20 tool slots). Verify via the health endpoint `<app-url>/`. 4. **Profiling only (yes in Step 2):** give the app an Azure service principal as described in `app/README.md` (*Optional: live Azure features*), then run it with `flowx(command="profile", parameters={"source": "adf", ...})`. Once added, the `discover`, `convert`, `package`, and `migrate` skills run **entirely through the `flowx` MCP tool** (`flowx(command="…", parameters={…})`) — there is no venv, no `bootstrap.sh`, and no `.migration-venv` marker on this path. > **Note:** Prefer the `app/deploy_app.py` notebook in serverless Genie Code — the SDK works there, > whereas `databricks apps deploy` / `sync` (used by `deploy.sh`) need a CLI session (web terminal > or local machine). Only fall back to `deploy.sh` from a web terminal. If `deploy.sh` reports > `Error: please specify target`, the CLI attached to a stray `databricks.yml`; it already isolates > against this, so re-run it as-is. --- ## Path B — Local / Claude Code (virtual environment) The flowx code in `src/flowx/` depends on third-party packages (`pyyaml`, `databricks-sdk`, `sqlglot`); running it against a bare system Python fails with `ModuleNotFoundError`. Provision an isolated venv (created once and reused). ### Step B1 — Run the bootstrap script ```bash bash <plugin_dir>/scripts/bootstrap.sh # user said no in Step 2 bash <plugin_dir>/scripts/bootstrap.sh --source-deps=azure # user said yes in Step 2 ``` Where `<plugin_dir>` is the flowx plugin root (the directory containing `src/`, `skills/`, and `requirements.txt`). The script will: 1. Check that `python3`, `pip`, and the `venv` module are available. 2. Create the venv at `<plugin_dir>/.venv`. 3. Install `requirements.txt` into that venv using `pip`. With `--source-deps=azure`, also install flowx from this checkout with its `azure` extra (`pip install -e '<plugin_dir>[azure]'`), so the installed version matches the source tree. 4. Write the resolved interpreter path to `<plugin_dir>/.migration-venv` for the other skills. ### Step B2 — Handle a missing Python or pip If Python, pip, or the `venv` module are **not** available, the script prints a `WARNING:` block and exits non-zero **without** creating anything. Do **not** work around it — relay the warning and stop: > ⚠️ Python must be installed before I can set up the flowx environment. > > * On macOS: `brew install python`. > * On Debian/Ubuntu: `sudo apt-get install python3 python3-venv python3-pip`. > > Let me know once it's installed and I'll re-run setup. Re-run this setup skill after the user confirms Python and pip are installed. ### Step B3 — Confirm success and how to run Python code On success, the script writes the interpreter path to `<plugin_dir>/.migration-venv`. The phase skills run Python with that interpreter and `src/` on `PYTHONPATH`. Resolve it from the marker file: ```bash export PYTHONPATH="<plugin_dir>/src" PY="$(cat <plugin_dir>/.migration-venv)" "$PY" -m flowx.adapter inputs discover --source adf # or --source airflow ``` `$PY` resolves to `<plugin_dir>/.venv/bin/python` (on Windows, `<plugin_dir>\.venv\Scripts\python.exe`). ### Step B4 — (Optional) Run the MCP server locally To drive the phases through MCP tools locally (instead of the CLI), install the MCP server stack into the venv and register the stdio server with your MCP client: ```bash PY="$(cat <plugin_dir>/.migration-venv)" "$PY" -m pip install "mcp>=1.12,<2" "uvicorn>=0.30" "starlette>=0.40" PYTHONPATH="<plugin_dir>/src" "$PY" -m flowx.mcp # stdio (default) ``` ```json { "mcpServers": { "flowx": { "command": "<interpreter from .migration-venv>", "args": ["-m", "flowx.mcp"], "env": { "PYTHONPATH": "<plugin_dir>/src" } } } } ``` ## Output | Path | Artifact | Description | |---|---|---| | A (Genie Code) | `mcp-flowx` Databricks App | Hosts the phases as the single `flowx` MCP tool at `<app-url>/mcp`; **no venv** is created | | B (local) | venv at `<plugin_dir>/.venv` | Virtual environment with the installed dependencies | | B (local) | `<plugin_dir>/.migration-venv` | Marker file holding the resolved interpreter path | | B (local, optional) | local MCP server | `mcp` / `uvicorn` / `starlette` installed into the venv; run with `python -m flowx.mcp` | ## Examples - "Set up flowx" (auto-detects Genie Code vs. local) - "Deploy the flowx MCP server to Databricks / Genie Code" (Path A) - "Bootstrap flowx so I can run a migration locally" (Path B) - "I got a ModuleNotFoundError running discover — fix the environment" (Path B)