| name | flowfile-build-and-env |
| description | Recreate every Flowfile dev/build environment from scratch (backend, web frontend, Tauri desktop, Docker stack + kernel images, WASM, pip-installed unified mode) with exact version pins, Makefile target semantics, PyInstaller/sidecar staging, and master-key generation — use when setting up a fresh checkout, running `make all`/`make services`/`make build_tauri_*`, debugging "sidecar not found"/notarization/stale-lock/npm-peer-dep failures, or answering "what Python/Node/Poetry/Rust version do I need" and "which port does X run on". |
Flowfile: build & environment recreation
This skill tells you how to stand up every flavor of the Flowfile dev/build
environment and what each Makefile target actually does under the hood (as
opposed to what its name implies). It does not cover how to write or test
code inside an already-running environment.
When NOT to use this skill
- You already have a working dev environment and just want to start/stop
services, seed demo data, or drive the app for a manual check — see
flowfile-run-and-operate.
- You're picking pytest markers, writing new tests, or setting up
Docker-backed test fixtures (postgres/mysql/s3/kafka) — see
flowfile-testing-and-validation.
- You need to know what an env var or feature flag does at runtime
(
FLOWFILE_MODE, FEATURE_FLAG_AI, offload flags) rather than how to set
it up for a build — see flowfile-config-and-flags.
- You're debugging a broken flow/AI/node at runtime, not a broken build —
see
flowfile-debugging-playbook or flowfile-ai-subsystem.
- You need the core/worker/kernel contract or subsystem architecture, not
build mechanics — see
flowfile-architecture-contract.
1. Ground-truth version pins (as of 2026-07-03, v0.12.7)
| Tool | Required | Where pinned | What breaks if you ignore it |
|---|
| Python | >=3.10,<3.14 | pyproject.toml:28 (python = ">=3.10,<3.14") | poetry install refuses to resolve on Python 3.9 or 3.14+; CI matrix tests 3.10–3.13 on Ubuntu, 3.11 on macOS |
| Poetry | 1.8.x (repo uses 1.8.4) | poetry.lock line 1 (@generated by Poetry 1.8.4); kernel_runtime/Dockerfile:25 pins POETRY_VERSION=1.8.4 | The Makefile's stale-lock self-heal runs poetry lock --no-update, a 1.x-only flag removed in Poetry 2.x. Poetry 2.x will error on install_python_deps/force_lock. Check with poetry --version before troubleshooting a "weird" lock failure. |
| Node | 20+ (CI runs 20 for build/publish workflows, 22 for the test.yaml web job; no engines field, no .nvmrc anywhere in the repo) | .github/workflows/*.yml node-version: lines | Nothing enforces a floor locally — an old Node silently fails deep inside Vite 6/Vitest 3 (frontend) or Vite 8/Vitest 4 (WASM) with an unhelpful syntax error. If npm install/npm run dev throws a cryptic parse error, check node --version first. |
| Rust (rustc + cargo) | stable toolchain; rust-version = "1.78" floor | flowfile_frontend/src-tauri/Cargo.toml:9; CI uses dtolnay/rust-toolchain@stable | rustc is required even to stage sidecars (not just to build the app) — tools/rename_sidecar.py shells out to rustc -vV to detect the target triple and raises RuntimeError if it's missing. Install via https://rustup.rs. |
| Tauri | v2 (tauri = "2" in Cargo.toml, @tauri-apps/cli ^2.0.0) | flowfile_frontend/src-tauri/Cargo.toml, flowfile_frontend/package.json | N/A — just don't mix v1 tutorials/docs in. |
| Polars | >=1.8.2, <1.40 — identical pin in root pyproject.toml and kernel_runtime/pyproject.toml, must be bumped together | pyproject.toml:37, kernel_runtime/pyproject.toml:13 | Bumping only one side drifts core/worker behavior from what runs inside kernel Docker containers; kernel containers additionally read their own poetry.lock at startup to report their actual resolved version. |
| PyInstaller | ^6.11.0, in the optional build dependency group | pyproject.toml:103-107 | Plain poetry install (no --with build) will not have pyinstaller — make build_python_services needs poetry install --with build first (the Makefile does this for you). |
| numpy | exactly 1.26.4 in both the dev group and kernel_runtime | pyproject.toml:118, kernel_runtime/pyproject.toml:15 | — |
Verify your machine matches:
python3 --version && poetry --version && node --version && npm --version && rustc --version
2. From-scratch recipes
(a) Backend dev (core + worker, no frontend)
poetry install
poetry run flowfile_core
poetry run flowfile_worker
Notes:
- Importing
flowfile_core has side effects: flowfile_core/flowfile_core/__init__.py calls validate_setup() then init_db() at module import time, which runs Alembic migrations against the live catalog SQLite DB. For any diagnostic script that just imports the package without wanting to touch the DB, set FLOWFILE_SKIP_STARTUP_MIGRATION=1 first.
FLOWFILE_MODE defaults to electron when unset. The embedded scheduler only starts if FLOWFILE_SCHEDULER_ENABLED is truthy.
- Local data lands under
~/.flowfile (internal storage) and your home dir (user data); override with FLOWFILE_STORAGE_DIR / FLOWFILE_USER_DATA_DIR.
poetry install --with build is only needed when you're going to run PyInstaller (§4) — skip it for plain backend dev, it just costs install time.
(b) Frontend web dev (no Tauri/Rust needed)
poetry run flowfile_core
cd flowfile_frontend
npm install
npm run dev:web
Notes (flowfile_frontend/vite.config.mjs):
root is src/renderer/; dev server listens on port 8080 with strictPort: true — it will refuse to start (not fall back to 8081/8082) if the port is taken, because Tauri's devUrl is hard-coded to http://localhost:8080 and a silent port-jump would break npm run dev.
- The
/api proxy strips the /api prefix when forwarding to core, and also rewrites FastAPI's absolute 307 trailing-slash redirects back under /api so client requests stay same-origin — a bug here manifests as requests that work in curl but 404 in the browser.
optimizeDeps.force: true means every cold start re-bundles dependencies (5–15s) — this is intentional, not a bug, to avoid a stale "504 Outdated Optimize Dep" error.
- There is no dev server on port 5173. The only
5173 reference in the repo is a leftover entry in core's CORS allowlist; if you see a tutorial mention 5173, it does not apply here.
(c) Full Tauri desktop build
Prereqs: Python 3.10–3.13 + Poetry 1.8.x + Node 20+ + a Rust toolchain (rustc must be on PATH, even just to stage sidecars).
make all
make all is exactly install_python_deps build_python_services rename_sidecars sign_sidecars build_tauri_app generate_key, run in that fixed order:
poetry install --with build (self-heals a stale lock first — see §5 gotcha 1)
poetry run build_backends → PyInstaller onedir builds of core + worker into services_dist/ (§4)
poetry run python tools/rename_sidecar.py → stages services_dist/ into flowfile_frontend/src-tauri/binaries/<name>-<host-triple> (§4b)
sign_sidecars → no-op unless macOS and APPLE_SIGNING_IDENTITY is set (§5)
cd flowfile_frontend && npm install && npm run build → npm run build = npm run lint && vue-tsc --noEmit && tauri build
generate_key → writes repo-root master_key.txt only if it doesn't already exist (§4c — note this file is dev convenience, not what the running app reads)
Day-to-day dev-mode desktop shell (skip the full release pipeline):
make services
cd flowfile_frontend && npm run dev
If you skip make services, the Tauri shell still launches but with no backend and no error at build time — sidecar resolution (flowfile_frontend/src-tauri/src/sidecar/mod.rs) just fails to find the binary at runtime. If the desktop app opens to a blank/non-functional UI, re-run make services.
Whenever you re-stage sidecars (re-run rename_sidecars after rebuilding core/worker), re-run npm run dev / rebuild — a stale cargo build can keep pointing at a binary that no longer matches.
Platform-specific one-shot builds (make build_tauri_win|_mac|_mac_arm|_mac_intel|_linux) call npx tauri build --target <triple> directly — unlike build_tauri_app they skip npm run lint and vue-tsc --noEmit and don't run npm install for you. Run npm install yourself first if node_modules isn't already present.
tauri.conf.json key facts: frontendDist: "../build/renderer", devUrl: "http://localhost:8080", beforeBuildCommand: "npm run build:renderer-only" (note: no lint/typecheck in this hook — that check happens earlier in npm run build), bundle targets: ["app","dmg","nsis","deb"], createUpdaterArtifacts: true. Sidecars ship via bundle.resources: ["binaries/**/*"], not Tauri's externalBin mechanism.
(d) Docker full stack
cp .env.example .env
docker compose up -d
Compose starts core, worker, and frontend only — kernel images are build-only profiles, not part of the default up:
docker compose --profile kernel build flowfile-kernel
docker compose --profile kernel build flowfile-kernel-ml
docker compose --profile kernel build flowfile-kernel-lite
Then point core at your local image (unset/empty env vars fall through to the published defaults edwardvaneechoud/flowfile-kernel-{base,ml,lite}:0.4.0):
FLOWFILE_KERNEL_IMAGE=flowfile-kernel-base:local
Other things worth knowing before you run this:
- Core mounts the host Docker socket (
/var/run/docker.sock) to manage kernel containers — this is a documented security tradeoff in docker-compose.yml, not an accident.
- Core + worker share a named volume
flowfile-internal-storage; user data bind-mounts to ./flowfile_data and ./saved_flows.
- The compose network is a fixed name (
flowfile-network) so kernel containers spun up dynamically via the Docker API can join it.
- Defaults are insecure on purpose for local dev (
admin/changeme, dev JWT/internal tokens) — override in .env for anything shared.
- There is no
secrets: block in compose. FLOWFILE_MASTER_KEY unset ⇒ the UI shows a first-run setup screen that prompts for one; env var always wins over a manually-added Docker secret file.
- Core and worker images are
python:3.12-slim; the Dockerfile installs psycopg2-binary/pymysql explicitly at image-build time even though those are dev-group-only in pyproject.toml — a plain poetry install --only main locally will NOT have Postgres/MySQL drivers.
(e) WASM dev (fully standalone — no backend)
cd flowfile_wasm
npm install --legacy-peer-deps
npm run dev
- WASM has its own
package.json/lockfile, independent of flowfile_frontend. It bumped to React 19 + @kanaries/graphic-walker@^0.5.0 to match the main frontend, but npm's strict peer-dep resolution still needs --legacy-peer-deps because GW's own transitive deps (headlessui, react-beautiful-dnd, react-leaflet…) haven't updated their declared peer ranges.
--legacy-peer-deps means npm will not auto-install peer deps for you — if npm install fails on a missing peer, add it to flowfile_wasm/package.json explicitly rather than dropping the flag (this has already bitten codemirror and styled-components once).
- Pyodide is loaded from a CDN at runtime, pinned to v0.27.7 (the last release with Polars support) — it is not an npm dependency, so
npm install won't show it. The pin also appears in the pyodide-smoke CI job and flowfile_wasm/tests/python/requirements.txt (which separately pins polars 1.18.0, pydantic 2.10.5, polars-expr-transformer 0.5.6 for the CPython-side test suite) — bump all of these together if you touch one.
- Dev server headers include COOP/COEP (
Cross-Origin-Opener-Policy: same-origin, Cross-Origin-Embedder-Policy: require-corp) — required for SharedArrayBuffer; anyone embedding the built package on another host page must set the same headers.
- Other scripts:
npm run build (app, vue-tsc --noEmit && vite build), npm run build:lib (BUILD_MODE=lib → dist/flowfile-editor.js, the published npm package), npm run test:run (Vitest one-shot, happy-dom env).
(f) pip-installed unified mode
pip install flowfile
flowfile run ui
Importing the flowfile package sets FLOWFILE_WORKER_PORT=63578 and FLOWFILE_SINGLE_FILE_MODE=1 as a side effect at import time (flowfile/flowfile/__init__.py) — do not import flowfile in the same process as a normally-configured core/worker pair, it will silently rewire the worker port out from under you. Other CLI verbs: flowfile run {ui|core|worker|flow <path>} (run flow forces local, non-offloaded execution and strips UI-only nodes), flowfile project {init|open|save}, flowfile seed-demo / remove-demo.
3. Makefile target reference (what each target truly does, not what its name implies)
| Target | Actual behavior |
|---|
all (default) | install_python_deps build_python_services rename_sidecars sign_sidecars build_tauri_app generate_key, in that order |
update_lock | poetry lock (resolves to latest allowed versions) |
force_lock | poetry lock --no-update (refresh lock file without bumping deps — Poetry 1.x-only flag) |
install_python_deps | Self-heals a stale lock (poetry check || poetry lock --no-update), then always poetry install --with build (includes PyInstaller even if you don't need it) |
build_python_services | Depends on install_python_deps; runs poetry run build_backends (PyInstaller, §4) |
rename_sidecars | poetry run python tools/rename_sidecar.py — needs rustc on PATH |
services | Convenience = build_python_services rename_sidecars |
sign_sidecars | bash tools/sign_macos_sidecars.sh — no-op unless macOS and APPLE_SIGNING_IDENTITY set; a Windows no-op branch is baked into the Makefile itself |
clean_dmg_mounts | macOS-only: force-detaches stale /Volumes/Flowfile* DMG mounts and deletes leftover rw.*.dmg temp images from a previous failed build (Tauri's bundle_dmg.sh runs set -e and dies on a name collision) |
build_tauri_app | Deps: clean_dmg_mounts sign_sidecars; then npm install && npm run build (lint + typecheck + tauri build) |
build_tauri_win / _mac / _mac_arm / _mac_intel / _linux | npx tauri build --target <triple> directly — skips lint/vue-tsc and npm install; mac variants still depend on clean_dmg_mounts sign_sidecars, win/linux have no deps |
measure_bundle | du -sh on services_dist/ and src-tauri/binaries/ — no build side effects, pure reporting |
test_built_services | Starts ./services_dist/flowfile_core and flowfile_worker in the background, sleeps 8s, curls /docs on 63578/63579, then POSTs /shutdown to both |
clean | Removes services_dist/ build/ flowfile_frontend/build/ flowfile_frontend/node_modules/ src-tauri/target/ src-tauri/binaries/ |
generate_key | Writes repo-root master_key.txt only if missing; chmod 600 on POSIX |
force_key | Same, but unconditional overwrite |
install_e2e | npm ci + npx playwright install chromium in flowfile_frontend |
test_e2e | npm run build:web; starts only poetry run flowfile_core (no worker!) + npm run preview:web (:4173); runs tests/web-flow.spec.ts with || true (failures are swallowed, not surfaced as a make failure); then stop_servers |
test_e2e_dev | Same, but uses npm run dev:web (:8080) instead of the preview server |
stop_servers | pkill -f flowfile_core, pkill -f vite (on Windows this is taskkill /F /IM python.exe and node.exe — kills every Python/Node process, not just this project's) |
test_coverage | pytest core, then worker, sequentially with --cov/--cov-append (avoids import collisions), then coverage report --show-missing |
stubs | Regenerates flowfile_frame/*_stub_generator.py output then runs ruff check --select F401 --fix on the generated .pyi files |
check_stubs | stubs + git diff --exit-code on the .pyi files — this is the CI drift gate, don't route around it |
bump-version VERSION=X.Y.Z | tools/bump_version.py — writes the version into all five synced manifests (see §1 note: use this, don't hand-edit) |
check-version | tools/check_version_sync.py — also run in release CI with --expect ${tag#v} |
4. PyInstaller pipeline
build_backends (poetry script → build_backends/build_backends/main.py) deletes services_dist/, then builds the worker first, then core, each from a generated .spec (written to repo root, deleted after the build) via pyinstaller --clean -y --dist ./services_dist.
Load-bearing details if you ever touch this pipeline:
- connectorx metadata hack: fabricates a fake
connectorx-0.4.3.dist-info plus a runtime hook monkeypatching importlib.metadata.version, because connectorx's real metadata doesn't survive freezing.
- Polars plugin packages (
polars_ds, polars_expr_transformer, polars_grouper, polars_simed, polars_distance) are collected with collect_submodules/collect_data_files/collect_dynamic_libs — a plain hiddenimport does not recurse subpackages or grab .abi3.so binaries. polars_simed/polars_distance look unused by static analysis but are transitive deps of pl_fuzzy_frame_match — removing them breaks startup. polars_ds is only ever imported lazily (inside shared/ml/trainers.py), so a naive static import scan misses it entirely.
- litellm, tokenizers, and
tiktoken_ext (a PEP 420 namespace package) all need explicit collection or the built binary silently can't find them at runtime.
- Excludes ~530MB of accidental ML stack (torch, transformers, sklearn, scipy, tensorflow) and pandas — pandas exclusion is intentional, it must never sneak into the bundle as a transitive dep.
- Output is
onedir (not onefile): a shared _internal/ directory plus two executables. combine_packages() merges the two builds' _internal/ trees into one to dedupe polars/pyarrow/numpy (roughly halves total bundle size).
build_backends_prd (main_prd.py) is not a production build despite the name — it's a local startup-time benchmarking harness with hardcoded personal paths. Don't treat it as a build target.
- Smoke-test the output with
make test_built_services before staging it into Tauri.
4b. Sidecar staging into src-tauri/binaries/<name>-<triple>
tools/rename_sidecar.py:
- Detects the host target triple via
rustc -vV (hard requirement — pass --triple <t> to cross-stage for a different target, which release CI does per matrix leg).
- Copies
services_dist/<name> → flowfile_frontend/src-tauri/binaries/<name>-<triple> (chmod 755).
- Copies
services_dist/_internal → binaries/_internal (delete-then-copytree; safe to re-run).
It errors clearly ("Run make build_python_services first") if services_dist/ is missing. Do not trust the script's own docstring about how build.rs wires things up — it claims build.rs symlinks binaries/_internal/ into target/; it does not. build.rs only bakes the compile-time TARGET triple into the FLOWFILE_TARGET_TRIPLE env var used by the sidecar path resolver. At runtime, flowfile_frontend/src-tauri/src/sidecar/mod.rs resolves the binaries directory directly — dev builds look under CARGO_MANIFEST_DIR/binaries, release builds under <resource_dir>/binaries — and _internal/ just needs to sit adjacent to the executable in that same directory, which the staging script already guarantees.
Master key generation: generate_key vs force_key
Both write a Fernet key to repo-root master_key.txt via cryptography.fernet.Fernet.generate_key(). generate_key (used by make all) is a no-op if the file already exists; force_key always overwrites. Neither is read by the running app — no runtime code path opens master_key.txt at that path. It exists purely as operator convenience for wiring a value into FLOWFILE_MASTER_KEY (env) or a Docker secret. The app's actual resolution order at startup is: FLOWFILE_MASTER_KEY env → Docker secret file /run/secrets/flowfile_master_key → electron mode auto-generates and stores one in local secure storage → otherwise it raises at startup telling you to set one. master_key.txt is gitignored alongside *.key, *.pem, .env.
5. Gotcha checklist (read this before filing a "build is broken" issue)
- Stale-lock self-heal assumes Poetry 1.8.x.
poetry lock --no-update (used by install_python_deps and force_lock) is a Poetry-1.x-only flag. If your local Poetry is 2.x, this step will error instead of silently fixing the lock — check poetry --version.
- Sidecar staging order is rigid and unenforced:
build_python_services → rename_sidecar.py (needs rustc) → sign → tauri build/dev. Running npm run dev with unstaged/stale binaries starts a shell with no backend and no build-time error.
build_tauri_win/_mac/_mac_arm/_mac_intel/_linux skip lint and vue-tsc and skip npm install — only build_tauri_app (and thus make all) runs the full npm run build check chain.
- macOS notarization requires a non-framework CPython build. A framework CPython (Homebrew, conda, macOS system Python,
actions/setup-python) makes PyInstaller bundle Python.framework with real symlinks; Tauri's resource copy step drops symlinks when staging binaries/**/* into the .app, which breaks the code-signature seal on that framework and fails notarization with "signature of the binary is invalid." Fix: use a python-build-standalone interpreter (e.g. via uv python install, UV_PYTHON_PREFERENCE=only-managed) for any build whose sidecars will be signed for distribution; verify with python -c "import sysconfig; print(sysconfig.get_config_var('PYTHONFRAMEWORK'))" — must print nothing/empty.
- Stale DMG mounts break
tauri build on macOS — a leftover /Volumes/Flowfile* mount from a previous failed build makes the bundler's bundle_dmg.sh (which runs under set -e) die. make clean_dmg_mounts (auto-run by the mac build targets) fixes this; run it by hand if you're not going through make.
make test_e2e starts core but not the worker, and swallows Playwright failures with || true — a red Playwright run will NOT fail this make target. Check the actual test output, not the exit code. stop_servers on Windows kills all python.exe/node.exe processes, not just this project's.
master_key.txt is not consumed by the running app — see §4b. Don't spend time debugging why setting it "doesn't work"; set FLOWFILE_MASTER_KEY (env) or the Docker secret file instead.
- DB drivers are dev-group-only in Poetry.
psycopg2-binary/pymysql live in [tool.poetry.group.dev]. A "prod-only" local install (poetry install --only main) will be missing Postgres/MySQL support; Docker images re-add them by hand at image-build time.
- Frontend and WASM are on different Vite/Vitest majors (frontend: Vite 6/Vitest 3; WASM: Vite 8/Vitest 4) — don't copy Vite/Vitest config snippets between the two packages without checking the target major's API.
strictPort: true + optimizeDeps.force: true on the frontend dev server are both intentional, not bugs: a taken port 8080 should fail loudly (Tauri's devUrl is hard-coded), and a forced re-optimize on every cold start avoids a worse failure mode (stale-dep 504s).
- A
v* git tag fires two release workflows at once (pypi-release.yml for PyPI, release.yaml for desktop installers); wasm-v* fires the separate npm WASM publish. Version bumps must go through make bump-version VERSION=X.Y.Z, not hand-edited manifests, or CI's check_version_sync.py --expect gate fails the release.
flowfile_wasm needs --legacy-peer-deps on npm install, and that flag means npm will not auto-resolve missing peers for you — if a new peer-dep error appears, add the package to flowfile_wasm/package.json explicitly rather than removing the flag.
- The PyPI wheel must contain the built frontend. Running
poetry build without first running npm run build:web and copying build/renderer/* into flowfile/flowfile/web/static/ produces a wheel where flowfile run ui has no UI at all. The copy step only exists inside pypi-release.yml — it is not part of any make target, so don't assume poetry build alone is release-ready.
6. Ports
| Port | What | Notes |
|---|
| 63578 | flowfile_core (FastAPI) | Also serves the bundled web UI in pip-installed unified mode (§2f) |
| 63579 | flowfile_worker (FastAPI) | |
| 8080 | Vite dev server (npm run dev:web) and nginx in Docker | Same port by design — Tauri's devUrl is hard-coded to it; strictPort: true so it never silently moves |
| 4173 | vite preview (used by make test_e2e) | |
| 5174 | WASM dev server (flowfile_wasm/vite.config.ts) | Standalone, no backend |
| 9999 | kernel container-internal uvicorn (EXPOSE 9999) | Not host-exposed directly |
| 19000–19999 | Host port range core maps kernel containers into | flowfile_core/flowfile_core/kernel/manager.py (_BASE_PORT=19000, _PORT_RANGE=1000) |
There is no dev server on 5173. If you see that port referenced in a tutorial or older doc, it does not apply to this repo — the only 5173 string in the codebase is a stale leftover in core's CORS allowlist.
Provenance and maintenance
Every fact above is dated 2026-07-03 (v0.12.7). Re-verify with these read-only commands before trusting this skill on a later checkout:
python3 tools/check_version_sync.py
sed -n '1,40p' pyproject.toml | grep -E 'version|python =|polars ='
grep -n 'rust-version' flowfile_frontend/src-tauri/Cargo.toml
head -1 poetry.lock
poetry --version
wc -l Makefile
grep -E '^[a-zA-Z_-]+:' Makefile
sed -n '1,70p' tools/rename_sidecar.py
cat flowfile_frontend/src-tauri/build.rs
grep -n 'devUrl\|frontendDist\|beforeBuildCommand\|beforeDevCommand\|"targets"\|resources' flowfile_frontend/src-tauri/tauri.conf.json
grep -n 'profiles\|image:\|FLOWFILE_KERNEL_IMAGE' docker-compose.yml
grep -n '_KERNEL_IMAGE.*_DEFAULT' flowfile_core/flowfile_core/kernel/manager.py
grep -n 'port\|COOP\|COEP' flowfile_wasm/vite.config.ts
cat flowfile_wasm/tests/python/requirements.txt
grep -rn 'EXPOSE\|--port' flowfile_core/Dockerfile flowfile_worker/Dockerfile kernel_runtime/Dockerfile
grep -n 'port' flowfile_frontend/vite.config.mjs flowfile_wasm/vite.config.ts
grep -rn 'master_key.txt' --include='*.py' .
If any of these disagree with this document, trust the command output and update this file — do not silently keep stale numbers.