Flowfile

Turn a plain-English description ("a node that runs on the kernel and does XGBoost predictions", "a node that trims whitespace", "an ML clustering node") into a correct single-file Flowfile custom node `.py` authored with the `node_designer` SDK (`from flowfile import node_designer as nd`, `CustomNodeBase`, `NodeSettings`/`Section`, `ColumnSelector`/`SingleSelect`/`NumericInput`/…). Use when a task says "generate/create/author/write a custom node", "make a node that does X", "a kernel node", "an sklearn/xgboost/lightgbm ML node", "a node with settings for …", when picking `environment="local"` vs `"kernel"` and `dependencies`, when reading settings inside `process()`, or when a generated node must pass the security scanner / bundle validation to install or publish. NOT for adding a built-in node type across core/frontend/frame/wasm — that is `flowfile-node-development`.

The editorial standard for Flowfile's docs site — the house style (register-2 voice rules), the persona-based nav map (which tab serves which arriving audience), the claim-type→source-of-truth verification index for fact-checking any docs statement against code, and the zero-drift examples contract (how tested .py and .yaml examples are structured, included via snippets, and auto-tested). Use when writing or reviewing any page under docs/, fact-checking a docs claim (node counts, Lite availability, API signatures, ports, providers, defaults), adding a worked example or tutorial, deciding where a new page belongs in the persona nav, or running an editorial/lint pass over docs changes.

How to author, wire, and verify the hand-drawn brand SVG diagrams on the Flowfile docs site — the exact file contract (viewBox, role + aria-label, defs→style→content), the shipped palette and typography ladders with exact hex values, the reusable component library (node chips, preview tables, arrow markers, decision diamonds, storage substrates, greyed insets), the concept-vs-technical register split and the one warm exception, the dark-mode technique against the two page canvases, the labeled-placeholder pattern for screenshots, and the redraw lessons the maintainer has already enforced. Use when creating or editing any SVG under docs/assets/images/, when a docs page needs a new concept or architecture diagram, when replacing or adding an IMAGE-PLACEHOLDER, when a diagram reads badly in dark mode, or when asked to draw, redraw, or fix any docs illustration.

How the Flowfile docs site (MkDocs + Material, the docs/ tree, mkdocs.yml nav) is organized, built by documentation.yml, and deployed to GitHub Pages; the repo-wide comment doctrine for code (and its two verbose exceptions); and the CLAUDE.md maintenance protocol across all 9 package guides. Use when writing or moving a page under docs/, editing mkdocs.yml's nav, linking between docs pages, regenerating docs/users/formulas/functions.md, debugging a Documentation workflow that's silently missing a page or won't build, deciding whether a line of code needs a comment, or updating any CLAUDE.md file after a feature lands.

Architecture and safe-extension guide for flowfile_core's /ai/* subsystem — the assist/copilot/planner surface map, the FEATURE_FLAG_AI router gate, the litellm lazy-import contract and its enforcing tests, BYOK provider resolution, per-process rate limiting, the prompt-log debugging runbook, the local llama.cpp model, and two maintainer-validated doctrines (prompt edits must be additive not subtractive; wrong-but-recoverable LLM payload shapes get normalized at the executor seam, not the schema or the prompt) — use when adding or debugging an AI route/agent/provider, when an agent tool call is being rejected or looping, when asked to "make the prompt more direct" or tighten AI/system prompts, when the LLM keeps emitting a wrong JSON shape for a tool call, or when investigating what a Flowfile LLM call actually saw/said.

The system map and load-bearing design contracts of Flowfile's core/worker/frontend/kernel/scheduler/shared architecture — who talks to whom, why core never materializes LazyFrames, the worker offload wire protocol, the $ffsec$ secrets format, dual node-execution state, node-hash caching discipline, and the pending/vision directions for execution and catalog decentralization. Use when onboarding to the codebase, deciding which service a new code path belongs in, debugging cross-service contract breaks (secrets, worker offload, kernel containers, scheduler), or evaluating whether a design change would foreclose the maintainer's decentralization vision.

Recreate every Flowfile dev/build environment from scratch (backend, web frontend, Tauri desktop, Docker stack + kernel images, WASM, pip-installed unified mode) with exact version pins, Makefile target semantics, PyInstaller/sidecar staging, and master-key generation — use when setting up a fresh checkout, running `make all`/`make services`/`make build_tauri_*`, debugging "sidecar not found"/notarization/stale-lock/npm-peer-dep failures, or answering "what Python/Node/Poetry/Rust version do I need" and "which port does X run on".

How changes to the Flowfile monorepo are classified, gated, versioned, and released — version-lockstep bump/check machinery, the stub and formula-docs drift gates, Alembic migration discipline, deliberate dependency pins (fastapi, polars), the v*/wasm-v* release-tag mechanics, real branch-protection state (ghost required checks, admin bypass), and the standing no-commit/no-stash agent working agreement. Use when bumping the app version, adding an Alembic migration, touching flowfile_frame's public API or the formula docs generator, editing a pinned dependency (fastapi, polars, litellm), preparing or reviewing a release/tag, wondering why a PR won't go green, or deciding whether an agent may run `git commit`/`git stash`.

An executable, decision-gated campaign to close the parity gap between a visual Flowfile flow, its exported Python code (Polars and FlowFrame), and the FlowFrame API — reproduce the baseline, work the divergence inventory, fix by the ranked solution menu, and promote through change control. Use when asked to "fix code generation", "export to Python is wrong", "flow result differs from generated code", "close the codegen xfails", "delete a stale xfail/XPASS", work on `flowfile_core/flowfile_core/flowfile/code_generator/`, or touch `test_code_generator*` parity tests.

Complete catalog of every Flowfile environment variable and runtime flag (FLOWFILE_MODE semantics, MutableBool live-flip flags, storage/database/catalog/AI/kernel/artifact/OAuth vars), which file:line reads each one, its default and truthy-parsing rule, and the documented-vs-actual drift between CLAUDE.md/.env.example/compose/docs — use when adding, changing, debugging, or auditing any FLOWFILE_*/env-var-driven behavior, when a config value "isn't taking effect", when writing a .env or docker-compose entry, or when asked "what env vars does Flowfile read" / "how do I flip a feature flag without restarting".

Symptom-to-cause triage playbook for Flowfile (core/worker/kernel/frontend/AI) — covers "no such table" DB cascades (two distinct causes), import-time Alembic migration corruption, silent axios/FastAPI 307 redirects in Docker, browser "CORS error" that is really an unhandled 500, stale-vs-rerunning node caching, "flow no longer in memory" 404s, Docker kernel container failures, Tauri desktop sidecar leaks, and AI agent misbehavior via the prompt log. Use when a flow run fails unexpectedly, a pytest run shows failures you didn't cause, the UI silently no-ops, the browser console says "CORS error", a node's result looks stale or keeps re-running for no reason, a scheduled/desktop run's logs seem to have vanished, a kernel (Python Script) node won't connect, quitting the desktop app leaves processes running, or an AI agent loops/hallucinates/misreferences columns — anytime you need to find root cause before writing a fix.

Chronicles 25+ major incidents in Flowfile's git history (worker/kernel deadlocks and races, reverted or self-reverted branches, Electron/Tauri packaging breakage, codegen correctness bugs, CI flakiness, security-fix lineage, in-flight save/open and flow-naming churn) with symptom, root cause, evidence commit/PR, and current status, plus a live/stalled-branch inventory and a verified-and-rejected list of already-settled debates — use BEFORE re-fighting a settled battle, before proposing a fix that touches worker transport, kernel lifecycle, sort/join/group_by codegen, flow save/open/naming, Tauri packaging or CSP, CI test ordering/coverage, or version/release tooling, and before trusting an old branch, a `git tag`, or a CLAUDE.md line as current truth without checking here first.

Deep dive into flowfile_frame — the Polars-LazyFrame-shaped Python API that builds an in-process flowfile_core FlowGraph as a side effect of every method call — covering the FlowFrame/Expr internals (_repr_str, _ff_repr, node-emission decision logic, method-injection decorators, lambda-source extraction), the generated-code contract with core's sandbox, exact lazy-vs-eager semantics, DB/cloud/catalog/Kafka/REST connectors, the native-vs-polars-code parity table, and the make-stubs/check-stubs .pyi pipeline. Use when writing or debugging flowfile_frame scripts, adding/changing a FlowFrame or Expr method, investigating why a `.collect()` triggered real I/O or a write happened before `.collect()`, deciding whether an operation will render as an editable node or an opaque code block in the Designer, chasing a `make check_stubs` CI failure, or asking "does read_api/read_database need flowfile_worker running".

Vue 3 renderer + Tauri 2 shell conventions for flowfile_frontend and flowfile_wasm — path aliases, Pinia store map, the axios trailing-slash 307 trap, the node-settings-by-glob-convention resolution system, VueFlow canvas wiring, desktop.ts as the sole Tauri boundary, the sidecar boot/readiness/shutdown ladder, the 19-file god-component TODO(refactor) policy, and WASM's explicit-run-only rule. Use when adding or editing a Vue component/view/store/route in flowfile_frontend, building a new node's settings UI, touching axios API wrappers or seeing unexplained 307s, changing anything under src-tauri/ (sidecar, lifecycle, capabilities), calling a Tauri/native API from renderer code, working in flowfile_wasm, or wiring the ShareDialog sharing UI onto a new connection view.

End-to-end runbook for adding or modifying a Flowfile node type across all four layers (flowfile_core settings/graph/template, flowfile_frontend UI registry, flowfile_frame Python API, flowfile_wasm parity) including the string-convention dispatch trap, the worker-does-all-fetching rule for network sources, and the stub/parity gates each layer enforces. Use when a task says "add a node", "new node type", "add a transform/reader/writer node", "wire up a node in the UI", "expose this as a FlowFrame method", "add this node to WASM", or when debugging a 404/AttributeError/HTTP 419 from `/update_settings/`, a settings drawer that silently fails to load, a `make check_stubs` CI failure, or a WASM node missing from the palette.

The maintainer's three long-horizon research bets for Flowfile (AI-native flow building, visual↔code round-trip, decentralized execution on a centralized catalog) with the verified existing assets each builds on, PR-sized first steps, falsifiable milestones, and the research discipline that turns a hunch into a merged change. Use when scoping ambitious/exploratory work, writing a design doc or spike, proposing a "big idea," choosing what to prototype next, evaluating whether an experiment is worth shipping, or asked "what is Flowfile trying to become" / "where is this going" / "research direction" / "north star."

How to run and operate Flowfile — every `flowfile` CLI verb and flag, the three headless flow-execution paths (CLI/PyInstaller/scheduler), local-dev vs single-process vs Docker service startup, the on-disk storage map (flows, catalog Delta tables, DB, logs, secrets, master key), and the state-inspection runbook. Use when starting core/worker/UI, running a flow headlessly or on a schedule, asking "where does Flowfile store X on disk," debugging why a flow/schedule/run didn't produce output, choosing FLOWFILE_MODE or FLOWFILE_SINGLE_FILE_MODE, or importing `flowfile`/`flowfile_core` in a script and needing to avoid mutating the live catalog DB.

Exact per-package pytest/vitest/playwright commands, the 6 pytest markers and which need Docker, the test_utils Docker fixture matrix, the shared-test-DB isolation model and its failure modes, xfail/skip discipline, and coverage/CI test-matrix mechanics for the Flowfile monorepo. Use when running or writing tests, diagnosing "no such table" or phantom test failures, deciding whether a change is "validated," seeing XPASS in test output, wiring a new Docker-backed fixture, or asking "which suite proves this change works."