weft-node-authoring

# Writing a custom node Tangle reads the dispatch protocol and [the review] checklist. The `node-smith` subagent, and an expert taking the hand, read the manual that follows them. Three terms hold across the file: - [the contract] is the node's typed interface, and Tangle designs it: one job in a sentence; every input port (name, type, required or optional, and `accepts` only when a wire would be a mistake) and every output port (name, type); the service it wraps, if any; anything the surrounding program depends on (a form schema, a trigger registration, infra). Every value the node takes from the graph is its own input port. A `List` or `JsonDict` input whose elements would come from separate wires is the wrong shape: a list literal cannot hold a wire, so every program would need a `List` node in front of it just to feed the port. (A port that genuinely takes a list of things, like `LlmInference`'s `tools`, is fine: the program builds it with the `List` node, never with Python.) An open-ended set of values (a query's parameters, a template's holes) is declared with `canAddInputPorts` and read with `ctx.inputs.custom()`. `PostgresExecuteQuery` is the pattern to copy, and any node's own `features` block (`weft describe-nodes --node <Type> --compact`) says whether it carries the flag, so you confirm there rather than trusting a name you remember. - [the report] is what the node-smith hands back, and the only thing Tangle sees of its work. - [the tiers] are the three test tiers a node carries: `basic` (no external world at all), `fake` (a stubbed client), `live` (the real service, real credentials, real money). `weft test-node <Type>` runs `basic` and `fake`, fast and free. Nobody but the user runs `live`: the node-smith writes those tests, names the service and declares any fixtures the test cannot self-provide, and the user runs them later, with consent, through `/weft-live-test`. Live tests do not cover infra nodes for now (a live test needs a service with a stored connection): an infra node is tested inside a real program, with the carving and fake-value tools of the `weft-sdp` skill (`--from`, `--emit`, `--target`). ## The dispatch protocol (Tangle) A node is missing only after the catalog says so: `weft describe-nodes --list` for the whole vocabulary, then `weft describe-nodes --node <Type> --compact` on anything close. If the catalog already holds a node that does the job, you go straight to the weft code. Otherwise: 1. **Design [the contract].** The node-smith implements it and never invents it. One job per node, and the permissions it needs are fixed by that job: when the permission would depend on WHICH input arrives (a post that needs one scope for a person and another for a company page), that is two nodes, each declaring its own `requiresScopes`, never one node with the rule in prose. Every value a node emits on a port is at most 100 KB (the wire limit, see the manual's "The wire limit"); a contract whose output could be bigger (page text, a long listing, a file's bytes) says how the node bounds it, or hands the bytes to storage and emits the file value. 2. **Dispatch one node-smith per package.** [the brief] is [the contract] plus the project context the node-smith cannot see (what [stage] this node feeds, what the upstream types are). Nodes that share a package (one service's access node and the nodes that use it, `nodes/linkedin/`) go to ONE node-smith with every contract in the brief, because two smiths writing into one package trip over each other's half-written files and one of them ends up creating a `package.toml` over the other's folder. Packages that share nothing go out in parallel, one node-smith each; nodes that depend on each other's types go out in sequence. 3. **Run [the review]** on [the report] against the checklist below. A report that fails goes back as a redispatch whose brief carries the previous attempt's folder, the specific finding (never "do better"), and what to keep. You never fix the node-smith's node yourself unless the fix is one line and obvious, because the next dispatch needs to know the pattern anyway. If you catch yourself editing the node-smith's `mod.rs` or `tests.rs`, stop and write: "Wait. Redispatch." Then write the finding into the brief. 4. **Wire it.** With the node green and in the catalog, read its `metadata.json` once more as delivered, and write the weft code. A redispatch that comes back with the same finding gets the finding restated in one sentence and nothing else. The third identical failure comes back to the user as "this contract is not landing, here is what I suspect", because by then the blocker is probably real. If [the report] is blocked, you judge the blocker: a real impossibility comes back to the user as "this cannot be done honestly, here is the closest shape"; a soft blocker (missing docs, rig limits) goes back out with what you know. ### [the review] checklist You never trust a report you can re-verify for the cost of one command. If you catch yourself accepting the quoted test output in [the report] as the verdict, stop and write: "Wait. My run is the verdict." Then run the command. **Re-verify first, always:** - Re-run `weft test-node <Type>` yourself. A report that claimed green and runs red is redispatched with the dishonesty named as the finding. - Diff the delivered `metadata.json` against the port list in [the report]. A port renamed or dropped between report and file is redispatched. - Read every test and ask: how would this test fail? A test with no answer (runs the node, ignores the result, asserts nothing about the outputs) is not a test, whatever its name says. - `weft validate --file src/main.weft < src/main.weft` still passes with the node in the catalog, and `weft describe-nodes --node <Type> --compact` succeeds. The folder sits beside the module that uses it under `src/`, or under `nodes/` when shared; never inside `nodes/base_catalog/`. **Then check [the contract] and the body:** - No port renamed, added, or dropped; the one job is still the one job; no assembled `List` or `JsonDict` input. - A skim of `mod.rs`: no fallbacks, no swallowed errors, no retry loops, no orchestration inside the body; failures are loud. - **Every loop and every long wait watches `ctx.cancellation()`.** Stopping is cooperative: nothing kills a node mid-call, so a loop with no cancellation arm is a node that cannot be stopped by anything, and the only sign is a run that never ends. - The `live` tests are written, with the service named and fixtures declared, and [the report] names them as not run. **The half-arsing catalog.** Each of these fails [the review] and goes back as a redispatch with the specific finding: - **smoke-only**: one test that runs the node once and asserts nothing. - **happy-path-only**: the error paths (the loud `node_bail!` failures) are never exercised. - **no closure test**: nothing covers an optional input arriving closed. - **weakened assertions**: the test checks that an output exists, not that it holds the expected value. - **swallowed in the test**: patterns like `if let Err(_) = ... {}` that pass on failure. - **coverage gap against the contract**: count the tests against the ports: every port in [the contract] needs a test that fails if its behavior breaks, and a port with none is the finding. - **live tests missing or hollow**: [the contract] names a service but there is no `NodeTest::live` entry for it, or the entry declares no service and no fixtures. - **stale versions**: an API or dependency version taken from memory or an old example instead of the service's current docs, or one the service no longer serves; the rule is under deps.toml. - **empty rig**: `tests()` returns an empty vec, or `tests.rs` does not exist, and [the report] did not say so. An access node is the one exception: its body is the `access_node!` macro, there is nothing of the smith's to test, and it ships with no `tests.rs` at all. - **flaky-dismissed**: an intermittently failing test waved off as flaky instead of chased to its race. A race in the node is the node's bug; a test made tolerant of it (a retry, a sleep, a longer timeout) fails [the review] on both counts. - **body smells**: `.ok()` discarding an error, a default value standing in for a missing input, a retry loop, orchestration inside the node, a database client or a process opened fresh on every run where `ctx.shared` would hold one. - **an infra output left unbaked**: an infra node's output that holds as long as the infra does (an address, a connection, a handle) with no `"baked": true`, so every run that reads it runs the node to make it again. - **a dead end in an image**: a state an infra container can sit in (a dead pairing, a lost credential, a revoked session) with no button on its display that leaves it, so the user's only way out is restarting or terminating the infra. Every such state gets an action, offered in every state; the rule is under Infra node. - **a marker in an outbound payload**: a `__weft_<kind>__` wrapper handed to a provider, a bridge, a form spec or a live item, instead of the plain URL, `data:` URL or `{ url, mimeType, filename }` that consumer reads; the rule is under A file input. - **silent failure in an image**: a service inside an infra image that fails a step without writing a line to its log, or answers the node with a success when the thing asked for did not fully happen; the two rules are under Infra node. ## The manual A node does one thing: calls an API, transcribes audio, writes a row. It never orchestrates: looping, retrying, branching and waiting for a person are the graph's job. It never does plumbing (transport, credentials, acknowledgement protocols, subscriptions, retry bookkeeping are the language's). A new node goes beside the module that uses it under `src/` (`src/billing/stripe/` next to `src/billing/charge.weft`), or under `nodes/` when several parts of the program share it, and is usable by its `type` name as soon as it is there; the build compiles its Rust directly. Its body may only `use` the `weft` crate, the crates its package declares in `deps.toml`, and code inside its own package; a sibling package's code is never on its path. ## Anatomy A node is a directory (folder snake_case, `"type"` PascalCase, struct `<Type>Node`): ``` nodes/my_thing/ metadata.json the declared surface: ports, config, presentation mod.rs the Rust body, a Node trait impl deps.toml optional: extra cargo crates, OS packages, build env tests.rs optional: the node's own tests ``` A package is a directory with `package.toml` (`[package] name`, shared `[dependencies]`); members are auto-detected as immediate subdirs holding a `metadata.json`; shared `.rs` files at the package root are reached by members as `use super::<file>;`. A package root may hold a partial `metadata.json` of defaults every member inherits (key-by-key, member wins; `type`/`label`/`description` are never inherited). Never place any of this under `nodes/base_catalog/`: `weft catalog update` wipes it. Each package compiles as its own crate, so a project's node sees its own package's shared files and nothing under `nodes/base_catalog/`: you cannot `use` a stdlib helper such as `elevenlabs.rs`. If you need one function from a stdlib helper, copy it into your package's own shared file and say so in [the report]. If you need a whole capability, report it as a ctx feature the language is missing. `weft`, `tokio`, `serde`, `serde_json`, `async-trait`, `anyhow` and `tracing` are always available without declaring them; anything else, `uuid` included, goes in `deps.toml`. ## metadata.json Unknown keys are a loud parse error. Top level: | Key | Meaning | |---|---| | `type`, `label`, `description` | identity, required | | `tags`, `icon`, `color` | search and presentation | | `inputs` | one list for wired data and design-time config | | `outputs` | output ports | | `types` | named type declarations, e.g. `"ChatHistory": "List[ChatMessage]"` | | `features` | flags: `catchErrors` (the node reaches outside; weft adds the `error` output and catches the body's failures onto it), `pure` (the body does nothing outside its run beyond answering its caller and the run's own files: no network, connection, `ctx.run`, wait, tag, or storage beyond this run; a durable run starts it without waiting for the database and runs it again after a crash if it had sent nothing on and reads no stream. `Text`, `Switch` and `Reply` are pure, `HttpRequest` is not; an outside ctx call on a pure node fails, naming the flag), `isTrigger`, `canAddInputPorts` (an open-ended set of values arrives as ports the author declares inline; the body reads `ctx.inputs.custom()`), `canAddOutputPorts`, `optionalCustomInputs`, `customInputType`, `oneOfRequired`, `showDebugPreview`, `liveEndpoint` (the endpoint serving this infra node's display, see [The display](#the-display)), `castPorts`, `hidden`, `answersCaller` (`"whole"`, `"stream"` or `"end"`: this node answers a live caller the way `Reply`, `Stream` or `Close` does; any custom node that answers the caller declares it, so the compiler counts it), `liveConnection` (`"http"` or `"websocket"`: this trigger holds a caller, as `Route` and `Socket` do) | | `portsFromConfig` | ports derived from a config list: `{ "field", "matchInput", "specs": [{kind, keyField, catchAll?, addsInputs, addsOutputs}] }` | | `firesWith` | trigger only: EVERY field a firing can carry, name to weft type, `?` on the name for sometimes-present (`{"scheduledTime": "String", "caller?": "JsonDict"}`). Checked exactly: a firing missing a required field is refused, and so is one carrying a field you did not name | | `display` | inline render: `{ "kind": "media" \| "link", "output" \| "input": "<port>" }` | | `validate` | declarative rules: `{ "when": {...}, "then": {message, level: "structural"\|"runtime", field} }` | | `requires_infra`, `images`, `publishes` | infra nodes | | `service` | access nodes only, the connection recipe | | `accessApps` | project-shipped OAuth apps | Input entry: `name`, `type`, `required`, `requiredWhenWired`, `accepts`, `widget`, `default`, `label`, `placeholder`, `description`, `requiresScopes`, `requiresValues`. If an unwired input means something of its own (no `instance`: the program's own copy), mark it `requiredWhenWired`, not `required`: unwired, the node runs; wired, a wire that delivers nothing skips the node instead of falling back to that meaning. If an optional input needs one more permission when it is used, leave it out of `requiresScopes`, which every program using the node is asked for. In `run`, when the input is used and before `ctx.client(&account)`, copy `account.required_permissions()` (it already holds the declared ones), push the extra one, and pass the list to `account.with_required_permissions`. Opening then refuses a connection that lacks it, naming the permission. A `select` or `multiselect` widget's `options` are a closed list: a program writing anything else fails with `literal-not-an-option`. Keep it closed when your code matches on the value. When the list is only the ones you know of and a provider can add more (model ids, voices), add `"free_text": true` to the widget, and the options become suggestions. A `code` widget's `language` is one of `python`, `javascript`, `sql`, `json`; any other word fails to load. `json` also fits a `JsonDict` input (edited as JSON text); the others edit a `String`. A `number` widget with a whole `step` (`"step": 1`) takes whole numbers only: the compiler checks a written value and the runtime a wired one, so your body can cast it to an integer. Output entry: `name`, `type`, `description` (an output has no `required`), and on an infra node `baked` (see Infra node). A `validate` rule's `when` is a closed set of conditions, combined with `all`, `any` and `not`: the input and config checks (`input_satisfied`, `input_wired`, `output_wired`, `config_present`, `config_equals`, ...), plus the graph checks below. A rule never names another node's type: it selects nodes by a feature they declare, with `with: {feature: value}`. - `run_reaches` (`direction: "downstream"` or `"upstream"`, `with`): the run this node is part of holds a node with that feature. `Route` checks `{"answersCaller": true}` downstream; `Reply` checks `{"liveConnection": true}` upstream. - `downstream_of` (`with`): a node with that feature runs before this one along the wires (how `Reply` refuses to follow `{"answersCaller": ["stream"]}`). - `per_instance`: this node exists once per instance: marked `@per_instance`, with an `@instance_filled` field, or reading one of those. A group that receives a per-instance value on any input counts for everything inside it and everything reading its ports. A node that cannot be copied per instance (a trigger serving one public address) refuses it with a rule, and the rule can be conditional (`all` with a config check). - `input_names` (`port`, `names`): every name written in that input (a String, each String of a list, or each key of an object) is something of this program. `names: {"node": {}}` is a node, narrowed with optional `role: "infra"` or `"trigger"` and `per_instance: true/false`; `names: {"field": {}}` is a field written `node.field`, narrowed with optional `instance_filled: true/false`. Wrap it in `not` to refuse a bad name. A wired name is not known yet, so the rule does not fire on it. `{with}`, `{per_instance_reason}` and `{names}` in the message name what the check found (`{per_instance_reason}` says why, path included: "it reads 'bridge'", "it sits inside group 'work', which receives 'bridge'"). Copy a real rule from `Route`, `Reply` or `StartInfra` before writing your own. `required` is written only as `"required": true`, on an input the node cannot run without. You leave the key off every other input: absent already means optional, so `"required": false` says nothing and reads as though you meant something by it. `accepts` is the list of drivers the port takes, `["literal", "wire"]` when absent, and absent is right for almost every port. You write `["wire"]` only for a port that needs a real node (a provider, a stored file, an `Access` handle a consumer reads). Restricting a port a program could plausibly fill with a written value is a review finding. Two port kinds never carry the list: a `Bus`/`Generator` port is wire-only (the loader forces it), and a compiler-read port (the `portsFromConfig` list, the access picker) takes an inline typed value only by a fixed rule. A minimal, real example (the catalog's `Text`): ```json { "type": "Text", "label": "Text",