weft-node-authoring

# Writing a custom node Tangle reads the dispatch protocol and [the review] checklist. The `node-smith` subagent, and an expert taking the hand, read the manual that follows them. Three terms hold across the file: - [the contract] is the node's typed interface, and Tangle designs it: one job in a sentence; every input port (name, type, required or optional, and `accepts` only when a wire would be a mistake) and every output port (name, type); the service it wraps, if any; anything the surrounding program depends on (a form schema, a trigger registration, infra). Every value the node takes from the graph is its own input port. A `List` or `JsonDict` input whose elements would come from separate wires is the wrong shape: a list literal cannot hold a wire, so it forces a Python node whose whole body is `return {'params': [a, b]}`. An open-ended set of values (a query's parameters, a template's holes) is declared with `canAddInputPorts` and read with `ctx.inputs.custom()`. `PostgresExecuteQuery` is the pattern to copy, and any node's own `features` block (`weft describe-nodes --node <Type> --compact`) says whether it carries the flag, so you confirm there rather than trusting a name you remember. - [the report] is what the node-smith hands back, and the only thing Tangle sees of its work. - [the tiers] are the three test tiers a node carries: `basic` (no external world at all), `fake` (a stubbed client), `live` (the real service, real credentials, real money). `weft test-node <Type>` runs `basic` and `fake`, fast and free. Nobody but the user runs `live`: the node-smith writes those tests, names the service and declares any fixtures the test cannot self-provide, and the user runs them later, with consent, through `/weft-live-test`. ## The dispatch protocol (Tangle) A node is missing only after the catalog says so: `weft describe-nodes --list` for the whole vocabulary, then `weft describe-nodes --node <Type> --compact` on anything close. If the catalog already holds a node that does the job, you go straight to the weft code. Otherwise: 1. **Design [the contract].** The node-smith implements it and never invents it. One job per node, and the permissions it needs are fixed by that job: when the permission would depend on WHICH input arrives (a post that needs one scope for a person and another for a company page), that is two nodes, each declaring its own `requiresScopes`, never one node with the rule in prose. Every value a node emits on a port is at most 100 KB (the wire limit, see the manual's "The wire limit"); a contract whose output could be bigger (page text, a long listing, a file's bytes) says how the node bounds it, or hands the bytes to storage and emits the file value. 2. **Dispatch one node-smith per package.** [the brief] is [the contract] plus the project context the node-smith cannot see (what [stage] this node feeds, what the upstream types are). Nodes that share a package (one service's access node and the nodes that use it, `nodes/linkedin/`) go to ONE node-smith with every contract in the brief, because two smiths writing into one package trip over each other's half-written files and one of them ends up creating a `package.toml` over the other's folder. Packages that share nothing go out in parallel, one node-smith each; nodes that depend on each other's types go out in sequence. 3. **Run [the review]** on [the report] against the checklist below. A report that fails goes back as a redispatch whose brief carries the previous attempt's folder, the specific finding (never "do better"), and what to keep. You never fix the node-smith's node yourself unless the fix is one line and obvious, because the next dispatch needs to know the pattern anyway. If you catch yourself editing the node-smith's `mod.rs` or `tests.rs`, stop and write: "Wait. Redispatch." Then write the finding into the brief. 4. **Wire it.** With the node green and in the catalog, read its `metadata.json` once more as delivered, and write the weft code. A redispatch that comes back with the same finding gets the finding restated in one sentence and nothing else. The third identical failure comes back to the user as "this contract is not landing, here is what I suspect", because by then the blocker is probably real. If [the report] is blocked, you judge the blocker: a real impossibility comes back to the user as "this cannot be done honestly, here is the closest shape"; a soft blocker (missing docs, rig limits) goes back out with what you know. ### [the review] checklist You never trust a report you can re-verify for the cost of one command. If you catch yourself accepting the quoted test output in [the report] as the verdict, stop and write: "Wait. My run is the verdict." Then run the command. **Re-verify first, always:** - Re-run `weft test-node <Type>` yourself. A report that claimed green and runs red is redispatched with the dishonesty named as the finding. - Diff the delivered `metadata.json` against the port list in [the report]. A port renamed or dropped between report and file is redispatched. - Read every test and ask: how would this test fail? A test with no answer (runs the node, ignores the result, asserts nothing about the outputs) is not a test, whatever its name says. - `weft validate --file src/main.weft < src/main.weft` still passes with the node in the catalog, and `weft describe-nodes --node <Type> --compact` succeeds. The folder sits beside the module that uses it under `src/`, or under `nodes/` when shared; never inside `nodes/base_catalog/`. **Then check [the contract] and the body:** - No port renamed, added, or dropped; the one job is still the one job; no assembled `List` or `JsonDict` input. - A skim of `mod.rs`: no fallbacks, no swallowed errors, no retry loops, no orchestration inside the body; failures are loud. - **Every loop and every long wait watches `ctx.cancellation()`.** Stopping is cooperative: nothing kills a node mid-call, so a loop with no cancellation arm is a node that cannot be stopped by anything, and the only sign is a run that never ends. - The `live` tests are written, with the service named and fixtures declared, and [the report] names them as not run. **The half-arsing catalog.** Each of these fails [the review] and goes back as a redispatch with the specific finding: - **smoke-only**: one test that runs the node once and asserts nothing. - **happy-path-only**: the error paths (the loud `node_bail!` failures) are never exercised. - **no closure test**: nothing covers an optional input arriving closed. - **weakened assertions**: the test checks that an output exists, not that it holds the expected value. - **swallowed in the test**: patterns like `if let Err(_) = ... {}` that pass on failure. - **coverage gap against the contract**: count the tests against the ports: every port in [the contract] needs a test that fails if its behavior breaks, and a port with none is the finding. - **live tests missing or hollow**: [the contract] names a service but there is no `NodeTest::live` entry for it, or the entry declares no service and no fixtures. - **stale versions**: an API or dependency version taken from memory or an old example instead of the service's current docs, or one the service no longer serves; the rule is under deps.toml. - **empty rig**: `tests()` returns an empty vec, or `tests.rs` does not exist, and [the report] did not say so. An access node is the one exception: its body is the `access_node!` macro, there is nothing of the smith's to test, and it ships with no `tests.rs` at all. - **flaky-dismissed**: an intermittently failing test waved off as flaky instead of chased to its race. A race in the node is the node's bug; a test made tolerant of it (a retry, a sleep, a longer timeout) fails [the review] on both counts. - **body smells**: `.ok()` discarding an error, a default value standing in for a missing input, a retry loop, orchestration inside the node. - **a dead end in an image**: a state an infra container can sit in (a dead pairing, a lost credential, a revoked session) with no button on its display that leaves it, so the user's only way out is restarting or terminating the infra. Every such state gets an action, offered in every state; the rule is under Infra node. - **a marker in an outbound payload**: a `__weft_<kind>__` wrapper handed to a provider, a bridge, a form spec or a live item, instead of the plain URL, `data:` URL or `{ url, mimeType, filename }` that consumer reads; the rule is under A file input. - **silent failure in an image**: a service inside an infra image that fails a step without writing a line to its log, or answers the node with a success when the thing asked for did not fully happen; the two rules are under Infra node. ## The manual A node does one thing: calls an API, transcribes audio, writes a row. It never orchestrates (looping, retrying, branching, waiting for a person are the graph's job, and the engine gives journaling, resumability and cancellation for free) and never does plumbing (transport, credentials, acknowledgement protocols, subscriptions, retry bookkeeping are the language's). A new node goes in this project's `nodes/` folder and is usable by its `type` name as soon as it is there; the build compiles its Rust directly. Its body may only `use` the `weft` crate, the crates its package declares in `deps.toml`, and code inside its own package; a sibling package's code is never on its path. ## Anatomy A node is a directory (folder snake_case, `"type"` PascalCase, struct `<Type>Node`): ``` nodes/my_thing/ metadata.json the declared surface: ports, config, presentation mod.rs the Rust body, a Node trait impl deps.toml optional: extra cargo crates, OS packages, build env tests.rs optional: the node's own tests ``` A package is a directory with `package.toml` (`[package] name`, shared `[dependencies]`); members are auto-detected as immediate subdirs holding a `metadata.json`; shared `.rs` files at the package root are reached by members as `use super::<file>;`. A package root may hold a partial `metadata.json` of defaults every member inherits (key-by-key, member wins; `type`/`label`/`description` are never inherited). Never place any of this under `nodes/base_catalog/`: `weft catalog update` wipes it. Each package compiles as its own crate, so a project's node sees its own package's shared files and nothing under `nodes/base_catalog/`: you cannot `use` a stdlib helper such as `elevenlabs.rs`. If you need one function from a stdlib helper, copy it into your package's own shared file and say so in [the report]. If you need a whole capability, report it as a ctx feature the language is missing. `weft`, `tokio`, `serde`, `serde_json`, `async-trait`, `anyhow` and `tracing` are always available without declaring them; anything else, `uuid` included, goes in `deps.toml`. ## metadata.json Unknown keys are a loud parse error. Top level: | Key | Meaning | |---|---| | `type`, `label`, `description` | identity, required | | `tags`, `icon`, `color` | search and presentation | | `inputs` | one list for wired data and design-time config | | `outputs` | output ports | | `types` | named type declarations, e.g. `"ChatHistory": "List[ChatMessage]"` | | `features` | flags: `isTrigger`, `canAddInputPorts` (an open-ended set of values arrives as ports the author declares inline; the body reads `ctx.inputs.custom()`), `canAddOutputPorts`, `optionalCustomInputs`, `customInputType`, `oneOfRequired`, `showDebugPreview`, `liveEndpoint` (the endpoint serving this infra node's display, see [The display](#the-display)), `castPorts`, `hidden` | | `portsFromConfig` | ports derived from a config list: `{ "field", "matchInput", "specs": [{kind, keyField, catchAll?, addsInputs, addsOutputs}] }` | | `firesWith` | trigger only: EVERY field a firing can carry, name to weft type, `?` on the name for sometimes-present (`{"scheduledTime": "String", "caller?": "JsonDict"}`). Checked exactly: a firing missing a required field is refused, and so is one carrying a field you did not name | | `display` | inline render: `{ "kind": "media" \| "link", "output" \| "input": "<port>" }` | | `validate` | declarative rules: `{ "when": {...}, "then": {message, level: "structural"\|"runtime", field} }` | | `requires_infra`, `images`, `publishes` | infra nodes | | `service` | access nodes only, the connection recipe | | `accessApps` | project-shipped OAuth apps | Input entry: `name`, `type`, `required`, `accepts`, `widget`, `default`, `label`, `placeholder`, `description`, `requiresScopes`, `requiresValues`. Output entry: `name`, `type`, `description` (an output has no optionality). `required` is written only as `"required": true`, on an input the node cannot run without. You leave the key off every other input: absent already means optional, so `"required": false` says nothing and reads as though you meant something by it. `accepts` is the list of drivers the port takes, `["literal", "wire"]` when absent, and absent is right for almost every port. You write `["wire"]` only for a port that needs a real node (a provider, a history, an `Access` handle a consumer reads). Restricting a port a program could plausibly fill with a written value is a review finding. Two port kinds never carry the list: a `Bus`/`Generator` port is wire-only (the loader forces it), and a compiler-read port (the `portsFromConfig` list, the access picker) takes an inline typed value only by a fixed rule. A minimal, real example (the catalog's `Text`): ```json { "type": "Text", "label": "Text", "description": "Emit a literal string. Useful for prompts, labels, and config values.", "tags": ["literal", "string"], "icon": "Type", "color": "#64748b", "inputs": [ { "name": "value", "type": "String", "widget": { "kind": "textarea" }, "required": true, "label": "Value" } ], "outputs": [ { "name": "value", "type": "String" } ], "requires_infra": false } ``` ## mod.rs ```rust //! One doc line saying what the node does. use async_trait::async_trait; use weft::{ExecutionContext, Node, NodeManifest, WeftResult}; use weft::node::NodeOutput; #[derive(NodeManifest)] pub struct MyThingNode; #[cfg(feature = "node-tests")] mod tests; #[async_trait] impl Node for MyThingNode { #[cfg(feature = "node-tests")] fn tests(&self) -> Vec<weft::NodeTest> { tests::tests() } async fn run(&self, ctx: ExecutionContext) -> WeftResult<()> { let value: String = ctx.inputs.get("value")?; ctx.pulse_downstream(NodeOutput::new().set("out", value)).await } } ``` The `NodeManifest` derive embeds the sibling `metadata.json` at compile time (a missing or malformed file is a compile error). Ports are camelCase in JSON, and `ctx.inputs` is keyed by them. Config and wired inputs are one bag: `ctx.inputs.get::<T>("name")` returns the value however it arrived (wire, braces literal, assignment literal, or the declared `default`). You fail loudly, always: `WeftResult<()>`; `ctx.inputs.get(...)?` stamps its own errors; `node_bail!("message")` for conditions the node detects; `.node_err("context")?` wraps an external error with context. A node body never names a `WeftError` variant and never falls back to a default: the failure is recorded in the journal where the user reads it. If you catch yourself writing a fallback value, a `.ok()`, or a retry, stop and write: "Wait. Fail loudly." Then bail with the cause. You emit only through `ctx.pulse_downstream(NodeOutput::new().set(port, value))`; ports you did not emit are closed, which is the skip signal downstream. For user-added output ports use `ctx.fan_declared(...)`. Work that must not happen twice across a restart goes through `ctx.run(...)`, which replays the recorded result. **Stopping is cooperative, and a node that ignores it cannot be stopped.** Nothing kills a node mid-call: a cancelled execution (the caller left, a person pressed stop, a newer run stopped this one) only trips a flag, and the node is what acts on it. So anything that does not return promptly by itself watches that flag and returns when it trips: ```rust let cancel = ctx.cancellation(); loop { // ... one pass of the work ... tokio::select! { _ = tokio::time::sleep(interval) => {} _ = cancel.cancelled() => return Ok(()), } } ``` Every loop that waits, every long external call, every read that could block for minutes. A node that loops without it keeps running after everything that asked for it is gone, and the only sign is a run that never ends. If you catch yourself writing a loop with no cancellation arm, stop and write: "Wait. Nothing else can stop this." Then add the arm. Any node can stop other runs of its project from inside its own body, and two ctx calls are the whole of it (`TagRun` and `StopTagged` are thin wrappers over exactly these, with no privilege yours lacks): `ctx.tag_execution([tag, ...]).await?` puts tags on this run; `ctx.stop_tagged(tag, StopSelf::Keep).await?` stops every older run of the project carrying the tag, waiting ones included (a run parked on a person or a timer never wakes); `StopSelf::Include` stops this run too. Tag first, then stop: a stop only reaches runs that put the tag on before this one did, so when two runs race, the later one survives. Both calls are safe to re-run after a crash (a repeated tag keeps its place in the order, a repeated stop finds its targets already ended), so neither goes through `ctx.run`. A tag is `[A-Za-z0-9_-]{1,64}`; the ctx refuses anything else before writing. In the `fake` tier nothing is stopped: