crawfish-pipeline-model

name	crawfish-pipeline-model
description	The Crawfish pipeline mental model — Source → Filter → Batch → Aggregator → Router → Sink, and the Definition directory contract. Load when designing a pipeline or deciding between batch fan-out, an aggregator reduce, a router branch, or a refine loop.
allowed-tools	Read, Grep

The Crawfish pipeline mental model

A Crawfish project is agents that do bulk work over your data. The pipeline shape:

Source → Filter → Batch (fan-out) → Aggregator (reduce) → Router (branch) → Sink

Each node has one job:

• Source — produces the items to work over (fluid-source fan-out originates taint).
• Filter — drops items before fan-out (cheap pre-pass).
• Batch (fan-out) — runs the Definition once per item, independently. The workhorse.
• Aggregator (reduce) — folds many per-item outputs into one. Taint is the union — the fold is tainted if any input was (security rule 9).
• Router (branch) — branches on a label. A model-derived label may gate whether a consequential action fires, never choose among distinct consequential sinks.
• Sink — the consequential write. Its target is static-only (security rule 2); it fires only against a frozen (eval-mode) Definition (rule 7).

The Definition directory contract

A Definition is a directory (see docs/reference/definition.md). One line each:

• definition.py — typed IO (Parameter, Flow.STATIC vs fluid) + team shape (lead / coordination). The spine's primary surface.
• instructions.md — the lead agent's prompt (system prompt).
• agents/*.md — subagents; each is front-matter (role, delegates_to, tools, model) over a markdown body.
• tools/*.py — host-side callables; the callable name = the filename stem. Taint-aware IO.
• mcp/*.py — MCPConnections; auth is an env-var reference, never an inline value.
• policies/*.py — module-level Policy objects (budgets, retries, guards).
• skills/*.md — skill assets bundled with the Definition.
• fixtures/ — recorded inputs/outputs for deterministic craw test.

For the file-by-file authoring rules, see the crawfish-authoring playbook skills (the progressive-disclosure siblings shipped in this same plugin bundle).

Decision guide — which shape do I reach for?

Goal	Reach for	Why
Per-item independent work over many items	Batch fan-out	each item is processed alone; no shared state
Combine many per-item results into one	Aggregator (reduce)	a fold/vote/summary; remember taint is the union
Take different action depending on a label	Router (branch)	the label gates; it never chooses among sinks
Iterate toward a goal or bound	Refine loop	craw refine --until <goal/bound>; feedback stays fluid + tainted

Worked example — triage many tickets. A Source yields open tickets. A Batch fan-out runs the triage-bot Definition once per ticket (the ticket_body is fluid, the project is static). A Router branches on the model's severity label to decide whether to open an issue (gating a consequential action) — it does not choose which repo (that target is a static parameter). A Sink opens the GitHub issue against the static target, under the idempotency key derived from static config + the item id.

Coordination shapes

A Definition's team shape (coordination in definition.py):

• single — one agent does the whole task.
• lead — a lead agent delegates to named subagents (agents/*.md).
• sequential — agents run in a fixed pipeline order, each handing to the next.

Pick single for a one-shot transform, lead when work splits into specialist roles, and sequential when stages must run in order.