flowfile-build-and-env

name

flowfile-build-and-env

description

Recreate every Flowfile dev/build environment from scratch (backend, web frontend, Tauri desktop, Docker stack + kernel images, WASM, pip-installed unified mode) with exact version pins, Makefile target semantics, PyInstaller/sidecar staging, and master-key generation — use when setting up a fresh checkout, running `make all`/`make services`/`make build_tauri_*`, debugging "sidecar not found"/notarization/stale-lock/npm-peer-dep failures, or answering "what Python/Node/Poetry/Rust version do I need" and "which port does X run on".

Flowfile: build & environment recreation

This skill tells you how to stand up every flavor of the Flowfile dev/build environment and what each Makefile target actually does under the hood (as opposed to what its name implies). It does not cover how to write or test code inside an already-running environment.

When NOT to use this skill

• You already have a working dev environment and just want to start/stop services, seed demo data, or drive the app for a manual check — see flowfile-run-and-operate.
• You're picking pytest markers, writing new tests, or setting up Docker-backed test fixtures (postgres/mysql/s3/kafka) — see flowfile-testing-and-validation.
• You need to know what an env var or feature flag does at runtime (FLOWFILE_MODE, FEATURE_FLAG_AI, offload flags) rather than how to set it up for a build — see flowfile-config-and-flags.
• You're debugging a broken flow/AI/node at runtime, not a broken build — see flowfile-debugging-playbook or flowfile-ai-subsystem.
• You need the core/worker/kernel contract or subsystem architecture, not build mechanics — see flowfile-architecture-contract.

---

1. Ground-truth version pins (as of 2026-07-03, v0.12.7)

Tool	Required	Where pinned	What breaks if you ignore it
Python	>=3.10,<3.14	pyproject.toml:28 (python = ">=3.10,<3.14")	poetry install refuses to resolve on Python 3.9 or 3.14+; CI matrix tests 3.10–3.13 on Ubuntu, 3.11 on macOS
Poetry	1.8.x (repo uses 1.8.4)	poetry.lock line 1 (@generated by Poetry 1.8.4); kernel_runtime/Dockerfile:25 pins POETRY_VERSION=1.8.4	The Makefile's stale-lock self-heal runs poetry lock --no-update, a 1.x-only flag removed in Poetry 2.x. Poetry 2.x will error on install_python_deps/force_lock. Check with poetry --version before troubleshooting a "weird" lock failure.
Node	20+ (CI runs 20 for build/publish workflows, 22 for the test.yaml web job; no engines field, no .nvmrc anywhere in the repo)	.github/workflows/*.yml node-version: lines	Nothing enforces a floor locally — an old Node silently fails deep inside Vite 6/Vitest 3 (frontend) or Vite 8/Vitest 4 (WASM) with an unhelpful syntax error. If npm install/npm run dev throws a cryptic parse error, check node --version first.
Rust (rustc + cargo)	stable toolchain; rust-version = "1.78" floor	flowfile_frontend/src-tauri/Cargo.toml:9; CI uses dtolnay/rust-toolchain@stable	rustc is required even to stage sidecars (not just to build the app) — tools/rename_sidecar.py shells out to rustc -vV to detect the target triple and raises RuntimeError if it's missing. Install via https://rustup.rs.
Tauri	v2 (tauri = "2" in Cargo.toml, @tauri-apps/cli ^2.0.0)	flowfile_frontend/src-tauri/Cargo.toml, flowfile_frontend/package.json	N/A — just don't mix v1 tutorials/docs in.

Verify your machine matches:

python3 --version && poetry --version && node --version && npm --version && rustc --version

---

2. From-scratch recipes

(a) Backend dev (core + worker, no frontend)

poetry install                    # main + dev groups (pytest, ruff, mkdocs, testcontainers...)
poetry run flowfile_core          # FastAPI on :63578
poetry run flowfile_worker        # separate terminal, :63579

Notes:

• Importing flowfile_core has side effects: flowfile_core/flowfile_core/__init__.py calls validate_setup() then init_db() at module import time, which runs Alembic migrations against the live catalog SQLite DB. For any diagnostic script that just imports the package without wanting to touch the DB, set FLOWFILE_SKIP_STARTUP_MIGRATION=1 first.
• FLOWFILE_MODE defaults to electron when unset. The embedded scheduler only starts if FLOWFILE_SCHEDULER_ENABLED is truthy.
• Local data lands under ~/.flowfile (internal storage) and your home dir (user data); override with FLOWFILE_STORAGE_DIR / FLOWFILE_USER_DATA_DIR.
• poetry install --with build is only needed when you're going to run PyInstaller (§4) — skip it for plain backend dev, it just costs install time.

(b) Frontend web dev (no Tauri/Rust needed)

poetry run flowfile_core          # start this FIRST — Vite proxies /api → :63578
cd flowfile_frontend
npm install
npm run dev:web                   # Vite dev server at http://localhost:8080

Notes (flowfile_frontend/vite.config.mjs):

• root is src/renderer/; dev server listens on port 8080 with strictPort: true — it will refuse to start (not fall back to 8081/8082) if the port is taken, because Tauri's devUrl is hard-coded to http://localhost:8080 and a silent port-jump would break npm run dev.
• The /api proxy strips the /api prefix when forwarding to core, and also rewrites FastAPI's absolute 307 trailing-slash redirects back under /api so client requests stay same-origin — a bug here manifests as requests that work in curl but 404 in the browser.
• optimizeDeps.force: true means every cold start re-bundles dependencies (5–15s) — this is intentional, not a bug, to avoid a stale "504 Outdated Optimize Dep" error.
• There is no dev server on port 5173. The only 5173 reference in the repo is a leftover entry in core's CORS allowlist; if you see a tutorial mention 5173, it does not apply here.

(c) Full Tauri desktop build

Prereqs: Python 3.10–3.13 + Poetry 1.8.x + Node 20+ + a Rust toolchain (rustc must be on PATH, even just to stage sidecars).

make all

make all is exactly install_python_deps build_python_services rename_sidecars sign_sidecars build_tauri_app generate_key, run in that fixed order:

1. poetry install --with build (self-heals a stale lock first — see §5 gotcha 1)
2. poetry run build_backends → PyInstaller onedir builds of core + worker into services_dist/ (§4)
3. poetry run python tools/rename_sidecar.py → stages services_dist/ into flowfile_frontend/src-tauri/binaries/<name>-<host-triple> (§4b)
4. sign_sidecars → no-op unless macOS and APPLE_SIGNING_IDENTITY is set (§5)
5. cd flowfile_frontend && npm install && npm run build → npm run build = npm run lint && vue-tsc --noEmit && tauri build
6. generate_key → writes repo-root master_key.txt only if it doesn't already exist (§4c — note this file is dev convenience, not what the running app reads)

Day-to-day dev-mode desktop shell (skip the full release pipeline):

make services                     # = build_python_services + rename_sidecars — MUST run before `npm run dev`
cd flowfile_frontend && npm run dev    # tauri dev; beforeDevCommand runs `npm run dev:web` for you

If you skip make services, the Tauri shell still launches but with no backend and no error at build time — sidecar resolution (flowfile_frontend/src-tauri/src/sidecar/mod.rs) just fails to find the binary at runtime. If the desktop app opens to a blank/non-functional UI, re-run make services.

Whenever you re-stage sidecars (re-run rename_sidecars after rebuilding core/worker), re-run npm run dev / rebuild — a stale cargo build can keep pointing at a binary that no longer matches.

Platform-specific one-shot builds (make build_tauri_win|_mac|_mac_arm|_mac_intel|_linux) call npx tauri build --target <triple> directly — unlike build_tauri_app they skip npm run lint and vue-tsc --noEmit and don't run npm install for you. Run npm install yourself first if node_modules isn't already present.

tauri.conf.json key facts: frontendDist: "../build/renderer", devUrl: "http://localhost:8080", beforeBuildCommand: "npm run build:renderer-only" (note: no lint/typecheck in this hook — that check happens earlier in npm run build), bundle targets: ["app","dmg","nsis","deb"], createUpdaterArtifacts: true. Sidecars ship via bundle.resources: ["binaries/**/*"], not Tauri's externalBin mechanism.

(d) Docker full stack

cp .env.example .env
# set FLOWFILE_MASTER_KEY, JWT_SECRET_KEY, FLOWFILE_INTERNAL_TOKEN, FLOWFILE_ADMIN_USER/PASSWORD
docker compose up -d
# frontend: http://localhost:8080 (nginx), core: :63578, worker: :63579

Compose starts core, worker, and frontend only — kernel images are build-only profiles, not part of the default up:

docker compose --profile kernel build flowfile-kernel        # -> flowfile-kernel-base:local
docker compose --profile kernel build flowfile-kernel-ml     # EXTRAS=ml build arg
docker compose --profile kernel build flowfile-kernel-lite   # SLIM_CONSTRAINTS=true build arg

Then point core at your local image (unset/empty env vars fall through to the published defaults edwardvaneechoud/flowfile-kernel-{base,ml,lite}:0.4.0):

# in .env or the shell running `docker compose up`
FLOWFILE_KERNEL_IMAGE=flowfile-kernel-base:local
# or per-flavour: FLOWFILE_KERNEL_IMAGE_BASE / _ML / _LITE

Other things worth knowing before you run this:

• Core mounts the host Docker socket (/var/run/docker.sock) to manage kernel containers — this is a documented security tradeoff in docker-compose.yml, not an accident.
• Core + worker share a named volume flowfile-internal-storage; user data bind-mounts to ./flowfile_data and ./saved_flows.
• The compose network is a fixed name (flowfile-network) so kernel containers spun up dynamically via the Docker API can join it.
• Defaults are insecure on purpose for local dev (admin/changeme, dev JWT/internal tokens) — override in .env for anything shared.
• There is no secrets: block in compose. FLOWFILE_MASTER_KEY unset ⇒ the UI shows a first-run setup screen that prompts for one; env var always wins over a manually-added Docker secret file.
• Core and worker images are python:3.12-slim; the Dockerfile installs psycopg2-binary/pymysql explicitly at image-build time even though those are dev-group-only in pyproject.toml — a plain poetry install --only main locally will NOT have Postgres/MySQL drivers.

(e) WASM dev (fully standalone — no backend)

cd flowfile_wasm
npm install
npm run dev                       # http://localhost:5174

• WASM has its own package.json/lockfile, independent of flowfile_frontend. It tracks React 19 and the same @kanaries/graphic-walker major as the main frontend; a plain npm install and a clean npm ci both resolve.
• On a peer-dep error, add the missing package to flowfile_wasm/package.json explicitly rather than reaching for --legacy-peer-deps — that flag stops npm auto-installing peers, which has already bitten codemirror and styled-components once.
• Pyodide is loaded from a CDN at runtime, pinned to v0.27.7 (the last release with Polars support) — it is not an npm dependency, so npm install won't show it. The pin also appears in the pyodide-smoke CI job and flowfile_wasm/tests/python/requirements.txt (which separately pins polars 1.18.0, pydantic 2.10.5, polars-expr-transformer 0.5.6 for the CPython-side test suite) — bump all of these together if you touch one.
• Dev server headers include COOP/COEP (Cross-Origin-Opener-Policy: same-origin, Cross-Origin-Embedder-Policy: require-corp) — required for SharedArrayBuffer; anyone embedding the built package on another host page must set the same headers.
• Other scripts: npm run build (app, vue-tsc --noEmit && vite build), npm run build:lib (BUILD_MODE=lib → dist/flowfile-editor.js, the published npm package), npm run test:run (Vitest one-shot, happy-dom env).

(f) pip-installed unified mode

pip install flowfile
flowfile run ui              # serves bundled web UI + core + worker on ONE port

Importing the flowfile package sets FLOWFILE_WORKER_PORT=63578 and FLOWFILE_SINGLE_FILE_MODE=1 as a side effect at import time (flowfile/flowfile/__init__.py) — do not import flowfile in the same process as a normally-configured core/worker pair, it will silently rewire the worker port out from under you. Other CLI verbs: flowfile run {ui|core|worker|flow <path>} (run flow forces local, non-offloaded execution and strips UI-only nodes), flowfile project {init|open|save}, flowfile seed-demo / remove-demo.

---

3. Makefile target reference (what each target truly does, not what its name implies)

Target	Actual behavior
all (default)	install_python_deps build_python_services rename_sidecars sign_sidecars build_tauri_app generate_key, in that order
update_lock	poetry lock (resolves to latest allowed versions)
force_lock	poetry lock --no-update (refresh lock file without bumping deps — Poetry 1.x-only flag)
install_python_deps	Self-heals a stale lock (poetry check || poetry lock --no-update), then always poetry install --with build (includes PyInstaller even if you don't need it)
build_python_services	Depends on install_python_deps; runs poetry run build_backends (PyInstaller, §4)
rename_sidecars	poetry run python tools/rename_sidecar.py — needs rustc on PATH
services	Convenience = build_python_services rename_sidecars
sign_sidecars	bash tools/sign_macos_sidecars.sh — no-op unless macOS and APPLE_SIGNING_IDENTITY set; a Windows no-op branch is baked into the Makefile itself
clean_dmg_mounts	macOS-only: force-detaches stale /Volumes/Flowfile* DMG mounts and deletes leftover rw.*.dmg temp images from a previous failed build (Tauri's bundle_dmg.sh runs set -e and dies on a name collision)
build_tauri_app	Deps: clean_dmg_mounts sign_sidecars; then npm install && npm run build (lint + typecheck + tauri build)
build_tauri_win / _mac / _mac_arm / _mac_intel / _linux

---

4. PyInstaller pipeline

build_backends (poetry script → build_backends/build_backends/main.py) deletes services_dist/, then builds the worker first, then core, each from a generated .spec (written to repo root, deleted after the build) via pyinstaller --clean -y --dist ./services_dist.

Load-bearing details if you ever touch this pipeline:

• connectorx metadata hack: fabricates a fake connectorx-0.4.3.dist-info plus a runtime hook monkeypatching importlib.metadata.version, because connectorx's real metadata doesn't survive freezing.
• Polars plugin packages (polars_ds, polars_expr_transformer, polars_grouper, polars_simed, polars_distance) are collected with collect_submodules/collect_data_files/collect_dynamic_libs — a plain hiddenimport does not recurse subpackages or grab .abi3.so binaries. polars_simed/polars_distance look unused by static analysis but are transitive deps of pl_fuzzy_frame_match — removing them breaks startup. polars_ds is only ever imported lazily (inside shared/ml/trainers.py), so a naive static import scan misses it entirely.
• litellm, tokenizers, and tiktoken_ext (a PEP 420 namespace package) all need explicit collection or the built binary silently can't find them at runtime.
• Excludes ~530MB of accidental ML stack (torch, transformers, sklearn, scipy, tensorflow) and pandas — pandas exclusion is intentional, it must never sneak into the bundle as a transitive dep.
• Output is onedir (not onefile): a shared _internal/ directory plus two executables. combine_packages() merges the two builds' _internal/ trees into one to dedupe polars/pyarrow/numpy (roughly halves total bundle size).
• build_backends_prd (main_prd.py) is not a production build despite the name — it's a local startup-time benchmarking harness with hardcoded personal paths. Don't treat it as a build target.
• Smoke-test the output with make test_built_services before staging it into Tauri.

4b. Sidecar staging into src-tauri/binaries/<name>-<triple>

tools/rename_sidecar.py:

1. Detects the host target triple via rustc -vV (hard requirement — pass --triple <t> to cross-stage for a different target, which release CI does per matrix leg).
2. Copies services_dist/<name> → flowfile_frontend/src-tauri/binaries/<name>-<triple> (chmod 755).
3. Copies services_dist/_internal → binaries/_internal (delete-then-copytree; safe to re-run).

It errors clearly ("Run make build_python_services first") if services_dist/ is missing. Do not trust the script's own docstring about how build.rs wires things up — it claims build.rs symlinks binaries/_internal/ into target/; it does not. build.rs only bakes the compile-time TARGET triple into the FLOWFILE_TARGET_TRIPLE env var used by the sidecar path resolver. At runtime, flowfile_frontend/src-tauri/src/sidecar/mod.rs resolves the binaries directory directly — dev builds look under CARGO_MANIFEST_DIR/binaries, release builds under <resource_dir>/binaries — and _internal/ just needs to sit adjacent to the executable in that same directory, which the staging script already guarantees.

Master key generation: generate_key vs force_key

Both write a Fernet key to repo-root master_key.txt via cryptography.fernet.Fernet.generate_key(). generate_key (used by make all) is a no-op if the file already exists; force_key always overwrites. Neither is read by the running app — no runtime code path opens master_key.txt at that path. It exists purely as operator convenience for wiring a value into FLOWFILE_MASTER_KEY (env) or a Docker secret. The app's actual resolution order at startup is: FLOWFILE_MASTER_KEY env → Docker secret file /run/secrets/flowfile_master_key → electron mode auto-generates and stores one in local secure storage → otherwise it raises at startup telling you to set one. master_key.txt is gitignored alongside *.key, *.pem, .env.

---

5. Gotcha checklist (read this before filing a "build is broken" issue)

1. Stale-lock self-heal assumes Poetry 1.8.x. poetry lock --no-update (used by install_python_deps and force_lock) is a Poetry-1.x-only flag. If your local Poetry is 2.x, this step will error instead of silently fixing the lock — check poetry --version.
2. Sidecar staging order is rigid and unenforced: build_python_services → rename_sidecar.py (needs rustc) → sign → tauri build/dev. Running npm run dev with unstaged/stale binaries starts a shell with no backend and no build-time error.
3. build_tauri_win/_mac/_mac_arm/_mac_intel/_linux skip lint and vue-tsc and skip npm install — only build_tauri_app (and thus make all) runs the full npm run build check chain.
4. macOS notarization requires a non-framework CPython build. A framework CPython (Homebrew, conda, macOS system Python, actions/setup-python) makes PyInstaller bundle Python.framework with real symlinks; Tauri's resource copy step drops symlinks when staging binaries/**/* into the .app, which breaks the code-signature seal on that framework and fails notarization with "signature of the binary is invalid." Fix: use a python-build-standalone interpreter (e.g. via uv python install, UV_PYTHON_PREFERENCE=only-managed) for any build whose sidecars will be signed for distribution; verify with python -c "import sysconfig; print(sysconfig.get_config_var('PYTHONFRAMEWORK'))" — must print nothing/empty.
5. Stale DMG mounts break tauri build on macOS — a leftover /Volumes/Flowfile* mount from a previous failed build makes the bundler's bundle_dmg.sh (which runs under set -e) die. make clean_dmg_mounts (auto-run by the mac build targets) fixes this; run it by hand if you're not going through make.
6. make test_e2e starts core but not the worker, and swallows Playwright failures with — a red Playwright run will NOT fail this make target. Check the actual test output, not the exit code. on Windows kills / processes, not just this project's.

---

6. Ports

Port	What	Notes
63578	flowfile_core (FastAPI)	Also serves the bundled web UI in pip-installed unified mode (§2f)
63579	flowfile_worker (FastAPI)
8080	Vite dev server (npm run dev:web) and nginx in Docker	Same port by design — Tauri's devUrl is hard-coded to it; strictPort: true so it never silently moves
4173	vite preview (used by make test_e2e)
5174	WASM dev server (flowfile_wasm/vite.config.ts)	Standalone, no backend
9999	kernel container-internal uvicorn (EXPOSE 9999)	Not host-exposed directly
19000–19999	Host port range core maps kernel containers into	flowfile_core/flowfile_core/kernel/manager.py (_BASE_PORT=19000, _PORT_RANGE=1000)

There is no dev server on 5173. If you see that port referenced in a tutorial or older doc, it does not apply to this repo — the only 5173 string in the codebase is a stale leftover in core's CORS allowlist.

---

Provenance and maintenance

Every fact above is dated 2026-07-03 (v0.12.7). Re-verify with these read-only commands before trusting this skill on a later checkout:

# App version + sync across all 5 manifests
python3 tools/check_version_sync.py

# Version pins
sed -n '1,40p' pyproject.toml | grep -E 'version|python =|polars ='
grep -n 'rust-version' flowfile_frontend/src-tauri/Cargo.toml
head -1 poetry.lock                              # Poetry version that generated the lock
poetry --version                                  # your local Poetry version — must match the lock's major

# Makefile target set hasn't drifted (line count + target names)
wc -l Makefile
grep -E '^[a-zA-Z_-]+:' Makefile

# Sidecar staging script still behaves as described
sed -n '1,70p' tools/rename_sidecar.py

# build.rs really doesn't symlink anything (docstring-vs-code drift check)
cat flowfile_frontend/src-tauri/build.rs

# Tauri config: devUrl / frontendDist / bundle targets
grep -n 'devUrl\|frontendDist\|beforeBuildCommand\|beforeDevCommand\|"targets"\|resources' flowfile_frontend/src-tauri/tauri.conf.json

# Docker kernel profiles + default image tags
grep -n 'profiles\|image:\|FLOWFILE_KERNEL_IMAGE' docker-compose.yml
grep -n '_KERNEL_IMAGE.*_DEFAULT' flowfile_core/flowfile_core/kernel/manager.py

# WASM peer-dep / port / Pyodide pin
grep -n 'port\|COOP\|COEP' flowfile_wasm/vite.config.ts
cat flowfile_wasm/tests/python/requirements.txt

# Ports (grep each service's own EXPOSE/listen/port config rather than trusting this table blindly)
grep -rn 'EXPOSE\|--port' flowfile_core/Dockerfile flowfile_worker/Dockerfile kernel_runtime/Dockerfile
grep -n 'port' flowfile_frontend/vite.config.mjs flowfile_wasm/vite.config.ts

# Master key: confirm nothing reads master_key.txt at runtime
grep -rn 'master_key.txt' --include='*.py' .

If any of these disagree with this document, trust the command output and update this file — do not silently keep stale numbers.