weft-node-authoring

# Writing a custom node Tangle reads the dispatch protocol and [the review] checklist. The `node-smith` subagent, and an expert taking the hand, read the manual that follows them. Three terms hold across the file: - [the contract] is the node's typed interface, and Tangle designs it: one job in a sentence; every input port (name, type, required or optional, and `accepts` only when a wire would be a mistake); every output port (name, type); every metadata port, if the node has one (name, type, and its value, written by hand); the service it wraps, if any; anything the surrounding program depends on (a form schema, a trigger registration, infra). Every value the node takes from the graph is its own input port. A `List` or `JsonDict` input whose elements would come from separate wires is the wrong shape: a list literal cannot hold a wire, so every program would need a `List` node in front of it just to feed the port. (A port that genuinely takes a list of things, like `LlmInference`'s `tools`, a `List[LlmTool]`, is fine: the program builds it with the `List` node, never with Python.) An open-ended set of values (a query's parameters, a template's holes) is declared with `canAddInputPorts` and read with `ctx.inputs.custom()`. `PostgresExecuteQuery` is the pattern to copy, and any node's own `features` block (`weft describe-nodes --node <Type> --compact`) says whether it carries the flag, so you confirm there rather than trusting a name you remember. - [the report] is what the node-smith hands back, and the only thing Tangle sees of its work. - [the tiers] are the three test tiers a node carries: `basic` (no external world at all), `fake` (a stubbed client), `live` (the real service, real credentials, real money). `weft test-node <Type>` runs `basic` and `fake`, fast and free. Nobody but the user runs `live`: the node-smith writes those tests, names the service and declares any fixtures the test cannot self-provide, and the user runs them later, with consent, through `/weft-live-test`. Live tests do not cover infra nodes for now (a live test needs a service with a stored connection): an infra node is tested inside a real program, with the carving and fake-value tools of the `weft-sdp` skill (`--from`, `--emit`, `--target`). ## The dispatch protocol (Tangle) A node is missing only after the catalog says so: `weft describe-nodes --list` for the whole vocabulary, then `weft describe-nodes --node <Type> --compact` on anything close. If the catalog already holds a node that does the job, you go straight to the weft code. Otherwise: 1. **Design [the contract].** The node-smith implements it and never invents it. One job per node, and the permissions it needs are fixed by that job: when the permission would depend on WHICH input arrives (a post that needs one scope for a person and another for a company page), that is two nodes, each declaring its own `requiresScopes`, never one node with the rule in prose. Every value a node emits on a port is at most 100 KB (the wire limit, see the manual's "The wire limit"); a contract whose output could be bigger (page text, a long listing, a file's bytes) says how the node bounds it, or hands the bytes to storage and emits the file value. 2. **Dispatch one node-smith per package.** [the brief] is [the contract] plus the project context the node-smith cannot see (what [stage] this node feeds, what the upstream types are). Nodes that share a package (one service's access node and the nodes that use it, `nodes/linkedin/`) go to ONE node-smith with every contract in the brief, because two smiths writing into one package trip over each other's half-written files and one of them ends up creating a `package.toml` over the other's folder. Packages that share nothing go out in parallel, one node-smith each; nodes that depend on each other's types go out in sequence. 3. **Run [the review]** on [the report] against the checklist below. A report that fails goes back as a redispatch whose brief carries the previous attempt's folder, the specific finding (never "do better"), and what to keep. You never fix the node-smith's node yourself unless the fix is one line and obvious, because the next dispatch needs to know the pattern anyway. If you catch yourself editing the node-smith's `mod.rs` or `tests.rs`, stop and write: "Wait. Redispatch." Then write the finding into the brief. 4. **Wire it.** With the node green and in the catalog, read its `metadata.json` once more as delivered, and write the weft code. A redispatch that comes back with the same finding gets the finding restated in one sentence and nothing else. The third identical failure comes back to the user as "this contract is not landing, here is what I suspect", because by then the blocker is probably real. If [the report] is blocked, you judge the blocker: a real impossibility comes back to the user as "this cannot be done honestly, here is the closest shape"; a soft blocker (missing docs, rig limits) goes back out with what you know. ### [the review] checklist You never trust a report you can re-verify for the cost of one command. If you catch yourself accepting the quoted test output in [the report] as the verdict, stop and write: "Wait. My run is the verdict." Then run the command. **Re-verify first, always:** - Re-run `weft test-node <Type>` yourself. A report that claimed green and runs red is redispatched with the dishonesty named as the finding. - Diff the delivered `metadata.json` against the port list in [the report]. A port renamed or dropped between report and file is redispatched. - Read every test and ask: how would this test fail? A test with no answer (runs the node, ignores the result, asserts nothing about the outputs) is not a test, whatever its name says. - `weft validate --file src/main.weft < src/main.weft` still passes with the node in the catalog, and `weft describe-nodes --node <Type> --compact` succeeds. The folder sits beside the module that uses it under `src/`, or under `nodes/` when shared; never inside `nodes/base_catalog/`. **Then check [the contract] and the body:** - No port renamed, added, or dropped; the one job is still the one job; no assembled `List` or `JsonDict` input. - A skim of `mod.rs`: no fallbacks, no swallowed errors, no retry loops, no orchestration inside the body; failures are loud. - **A wait of the node's own (a child process, a timer) goes through `ctx.cancellation().race(..)`**, never a hand `tokio::select!`. - The `live` tests are written, with the service named and fixtures declared, and [the report] names them as not run. **The half-arsing catalog.** Each of these fails [the review] and goes back as a redispatch with the specific finding: - **smoke-only**: one test that runs the node once and asserts nothing. - **happy-path-only**: the error paths (the loud `node_bail!` failures) are never exercised. - **no closure test**: nothing covers an optional input arriving closed. - **weakened assertions**: the test checks that an output exists, not that it holds the expected value. - **swallowed in the test**: patterns like `if let Err(_) = ... {}` that pass on failure. - **coverage gap against the contract**: count the tests against the ports: every port in [the contract] needs a test that fails if its behavior breaks, and a port with none is the finding. - **live tests missing or hollow**: [the contract] names a service but there is no `NodeTest::live` entry for it, or the entry declares no service and no fixtures. - **stale versions**: an API or dependency version taken from memory or an old example instead of the service's current docs, or one the service no longer serves; the rule is under deps.toml. - **empty rig**: `tests()` returns an empty vec, or `tests.rs` does not exist, and [the report] did not say so. An access node is the one exception: its body is the `access_node!` macro, there is nothing of the smith's to test, and it ships with no `tests.rs` at all. - **flaky-dismissed**: an intermittently failing test waved off as flaky instead of chased to its race. A race in the node is the node's bug; a test made tolerant of it (a retry, a sleep, a longer timeout) fails [the review] on both counts. - **an output with no bound**: a port whose value grows with what the service answers or what the node is given (a list of results, a page of text, a response body, a transcript) and nothing that keeps it under the 100 KB wire limit: no cap input or `LIMIT`, no paging, no stored file in its place. [the report] names each such port and its bound, and a `fake` test emits the largest value the bound allows; a port missing from that list is the finding. The rule is under The wire limit. - **body smells**: `.ok()` discarding an error, a default value standing in for a missing input, a retry loop, orchestration inside the node, a database client or a process opened fresh on every run where `ctx.pool` would hold one, weft's own work redone (a rule copied into Rust, a blank filter, a `format!` URL), heavy work (a big parse, image or audio work, compression, hashing a large file) done inline instead of in `tokio::task::spawn_blocking`. - **an infra output left unbaked**: an infra node's output that holds as long as the infra does (an address, a connection, a handle) with no `"baked": true`, so every run that reads it runs the node to make it again. - **a dead end in an image**: a state an infra container can sit in (a dead pairing, a lost credential, a revoked session) with no button on its display that leaves it, so the user's only way out is restarting or terminating the infra. Every such state gets an action, offered in every state; the rule is under Infra node. - **a marker in an outbound payload**: a `__weft_<kind>__` wrapper handed to a provider, a bridge, a form spec or a live item, instead of the plain URL, `data:` URL or `{ url, mimeType, filename }` that consumer reads; the rule is under A file input. - **silent failure in an image**: a service inside an infra image that fails a step without writing a line to its log, or answers the node with a success when the thing asked for did not fully happen; the two rules are under Infra node. ## The manual A node does one thing: calls an API, transcribes audio, writes a row. It never orchestrates: looping, retrying, branching and waiting for a person are the graph's job. It never does plumbing (transport, credentials, acknowledgement protocols, subscriptions, retry bookkeeping are the language's). A new node goes beside the module that uses it under `src/` (`src/billing/stripe/` next to `src/billing/charge.weft`), or under `nodes/` when several parts of the program share it, and is usable by its `type` name as soon as it is there; the build compiles its Rust directly. Its body may only `use` the `weft` crate, the crates its package declares in `deps.toml`, and code inside its own package; a sibling package's code is never on its path. ## Anatomy A node is a directory (folder snake_case, `"type"` PascalCase, struct `<Type>Node`): ``` nodes/my_thing/ metadata.json the declared surface: ports, config, presentation mod.rs the Rust body, a Node trait impl deps.toml optional: extra cargo crates, OS packages, build env tests.rs optional: the node's own tests ``` A package is a directory with `package.toml` (`[package] name`, shared `[dependencies]`); members are auto-detected as immediate subdirs holding a `metadata.json`; shared `.rs` files at the package root are reached by members as `use super::<file>;`. A package root may hold a partial `metadata.json` of defaults every member inherits (key-by-key, member wins; `type`/`label`/`description` are never inherited). Never place any of this under `nodes/base_catalog/`: `weft catalog update` wipes it. Each package compiles as its own crate, so a project's node sees its own package's shared files and nothing under `nodes/base_catalog/`: you cannot `use` a stdlib helper such as `elevenlabs.rs`. If you need one function from a stdlib helper, copy it into your package's own shared file and say so in [the report]. If you need a whole capability, report it as a ctx feature the language is missing. `weft`, `tokio`, `serde`, `serde_json`, `async-trait`, `anyhow` and `tracing` are always available without declaring them; anything else, `uuid` included, goes in `deps.toml`. ## metadata.json Unknown keys are a loud parse error. Top level: | Key | Meaning | |---|---| | `type`, `label`, `description` | identity, required | | `tags`, `icon`, `color` | search and presentation | | `inputs` | one list for wired data and design-time config | | `outputs` | output ports | | `metadataPorts` | ports whose value you write here by hand and no run computes, each `{ "name", "type", "description", "value" }` (`description` optional). A wire leaving one is a metadata wire: the compiler writes the value into the input it points at, as if typed there, and the wire never runs. Their names share one namespace with `inputs` and `outputs` | | `types` | named type declarations, e.g. `"ChatHistory": "List[ChatMessage]"` | | `features` | flags: `pure` (the body does nothing outside its run beyond answering its caller and the run's own files: no network, connection, wait, tag, or storage beyond this run; a durable run starts it without waiting for the database and runs it again after a crash if it had sent nothing on and reads no stream. `Text`, `Switch` and `Reply` are pure, `HttpRequest` is not; an outside ctx call on a pure node fails, naming the flag), `isTrigger`, `canAddInputPorts` (an open-ended set of values arrives as ports the author declares inline; the body reads `ctx.inputs.custom()`), `canAddOutputPorts`, `join` (the node takes several branches and carries on with whichever came, like `FirstInOrder`: every input you add is optional, it fires once all it waits on has arrived or closed, closes its own output when nothing came, and needs no `_should_flow`), `customInputType`, `customOutputType` (the type every added output takes, as `customInputType` is for added inputs; a variable one of the node's inputs uses, `T`, types each added output as that input), `pairsAddedPorts` (each input a program adds comes with an output of the same name, which the compiler makes and the program never writes; an added output nothing reads fails the build; declared beside `canAddOutputPorts`, as `Dispatch` does), `oneOfRequired`, `showDebugPreview`, `liveEndpoint` (the endpoint serving this infra node's display, see [The display](#the-display)), `castPorts`, `hidden`, `answersCaller` (`"whole"`, `"stream"` or `"end"`: this node answers a live caller the way `Reply`, `Stream` or `Close` does; any custom node that answers the caller declares it, so the compiler counts it), `liveConnection` (`"http"` or `"websocket"`: this trigger holds a caller, as `Route` and `Socket` do), `addedPortsAreDataKeys` (under Spelling) | | `portsFromConfig` | ports derived from a config list: `{ "field", "matchInput", "specs": [{kind, keyField, catchAll?, addsInputs, addsOutputs}] }` | | `firesWith` | trigger only: EVERY field a firing can carry, name to weft type, `?` on the name for sometimes-present (`{"scheduledTime": "String", "caller?": "JsonDict"}`). Checked exactly: a firing missing a required field is refused, and so is one carrying a field you did not name. On a node without `features.isTrigger`, or with a type weft cannot parse, the node fails to load | | `display` | inline render: `{ "kind": "media" \| "link", "output" \| "input": "<port>" }` | | `validate` | declarative rules: `{ "when": {...}, "then": {message, level: "structural"\|"runtime", field} }` | | `requiresInfra`, `images`, `publishes` | infra nodes | | `service` | access nodes only, the connection recipe | | `claimsRoute` | a node that claims a public address: `pathField` names the input holding the route pattern, and the optional `methodField` the input holding the HTTP method. Leave `methodField` out and the node serves every method; with it set, a program that leaves that input empty serves every method too. Naming an input the node does not declare fails the load | | `accessApps` | project-shipped OAuth apps, keyed by service name (`google`); what an entry may hold is below the table | | `placeholdersAreInputs` | a node whose added inputs are named inside one of its text settings (a query's `$userId`, a template's `{{name}}`): `{ "in": "<that setting>", "pattern": "<a regex whose group (?<name>...) is the input>" }`. The compiler then holds the two to each other, so every placeholder the setting holds must be an input of the node and every added input must appear in it, and a typo in a query fails the build instead of the first call. The pattern may also match text to skip (a string literal, a comment) through an alternative without the `name` group; those matches are ignored. Copy `Format`'s or `PostgresExecuteQuery`'s before writing your own | An `accessApps` entry holds `label`, `client_id` and the service's `registrationFields` that it marks `"secret": false` (a field is secret unless marked otherwise). The entry ships to everyone who uses the node, so a `client_secret` or a secret registration field in it fails the node's load. The load also fails when the key names no service in the project's catalog, or a service that does not sign in through OAuth, or a service that needs a secret field in every app. It fails too when `label`, `client_id` or a registration field the service does not mark `optional` is missing or empty, and when the entry has a key the service does not declare. ### Spelling The `type`, and any type you declare under `types`, is PascalCase (`WordCount`). Every key weft defines is camelCase (`requiresInfra`, `freeText`, `dependsOn`, `connectionOptional`), and so is every port, setting and record field you name, each field under `firesWith`, the names under each event's `fields` in a service's `events`, and each name of a value a connection stores (`imapHost`), both where a service declares it (`fields`, `captures`, `registrationFields`) and where an input asks for it (`requiresValues`). Two kinds of key are snake_case: an event topic (`drive_changes`) and a key under `accessApps` (`google`). They are snake_case because a node also writes them as values (a trigger picks a topic by name, and a node names its service as `"service": "google"`), and every value from a fixed list weft defines is snake_case (a widget's `kind`, a rule's `kind`). The `options` of a select you write keep whatever spelling the values have (`Africa/Abidjan`). A name somebody else chose keeps their spelling: an OAuth parameter under `authParams` (`access_type`), `client_id` in an `accessApps` entry, a field of a body you send a provider. A key weft used to spell in snake_case (`requires_infra`) fails the load, and the error names the new spelling (``requires_infra` is spelled `requiresInfra` now``); in a package root's defaults, the whole package is left out. A snake_case key weft never had is an unknown key: on a node it fails the load, and in a package root's defaults the root gets a warning instead and every member that inherits it fails to load. Any other name in the wrong casing loads, with a catalog warning: `weft describe-nodes --list` and `weft validate < src/main.weft` print it on stderr as `warning: <NodeType> (<path>): <message>` (a package root's defaults as `warning: package '<name>' defaults (<path>): ...`), and the graph view lists it as "node spelling warning". Clear every one on a node you write. If the ports a program adds to your node are named after data from outside weft (the captures in a route's path, a query's parameters, a template's ids), set `features.addedPortsAreDataKeys`. The compiler's `not-camel-case` warning then leaves the ports a program adds alone, and still checks the name the program gives the node. Input entry: `name`, `type`, `required`, `accepts`, `widget`, `default`, `label`, `placeholder`, `description`, `requiresScopes`, `requiresValues`, `keepsBlank`. A blank optional `String` arrives absent (`opt` gives `None`), so never filter blanks; `"keepsBlank": true` keeps one that means something. If an unwired input means something of its own (no `instance`: the program's