comfyui-workflow

# ComfyUI Workflow Submit workflows, monitor the queue, and retrieve outputs from a local ComfyUI server. This is the workhorse Skill; `comfyui-character` builds on top of it for character-consistent generation. ## Scope This Skill covers the **transport** between an agent and a local ComfyUI server. It does not edit workflows, train models, or invent prompts — those concerns live in sibling Skills. It assumes the user already has a workflow JSON they want to run (either a file the user wrote or one of the bundled `workflows/*.json` examples). ## Two equivalent entry points You can drive ComfyUI from this Skill in either of two ways. Pick whichever the host environment supports best. ### A. Python script (CLI-friendly, no MCP required) ```bash # Health probe python scripts/submit_workflow.py --probe # Submit a workflow python scripts/submit_workflow.py \ --workflow ./workflows/text-to-image.json \ --prompt "a tabby cat sleeping on a windowsill, morning light, photorealistic" \ --output-dir ./out # Check the queue without submitting python scripts/submit_workflow.py --queue # Download a specific image python scripts/submit_workflow.py --download \ --filename ComfyUI_00001_.png --output-dir ./out ``` The script lives at `skills/comfyui-workflow/scripts/submit_workflow.py` (relative to the Plugin root) and accepts `COMFYUI_URL` from the environment, falling back to `http://127.0.0.1:8188`. Full flag list in `references/api-reference.md`. ### B. stdio MCP server (no Python required) When the host agent is MCP-aware, the Plugin's `server.mjs` exposes three tools that mirror the Python script: | Tool | Mirrors | |---|---| | `submit_prompt` | `--workflow` invocation | | `check_queue` | `--queue` invocation | | `get_image` | `--download` invocation | The MCP server has zero npm dependencies. It is a 200-line stdio JSON-RPC server in plain Node. See `mcp.json` for the registration snippet and `server.mjs` for the source. ## Workflow contract ComfyUI accepts a JSON object whose top-level keys are node IDs and whose values are `{class_type, inputs}` records. When the user provides a workflow, **do not edit the structure** unless they ask. Two patterns are common: - A workflow file saved from the ComfyUI web UI: keys are numeric strings, `class_type` matches the node name in the editor, `inputs` reference other nodes by `[nodeId, outputIndex]`. - An "API format" workflow produced by ComfyUI's "Save (API Format)" menu: same shape, but with widget values inlined into `inputs`. Both work identically when POSTed to `/prompt`. The Plugin's `workflows/text-to-image.json` and `workflows/image-to-image.json` are in the API format. ## Overriding prompt values at submit time The bundled Python script understands a tiny templating convention so you do not have to rewrite the workflow JSON for every run. In a workflow, a `CLIPTextEncode` node with `inputs.text` starting with the literal string `__PROMPT__` will have that text replaced with the value of `--prompt` on the command line before submission. This is a no-op for nodes that do not opt in. ```json { "30": { "class_type": "CLIPTextEncode", "inputs": { "text": "__PROMPT__", "clip": ["4", 0] } } } ``` For workflows without that marker, the user must edit the workflow JSON directly. ## Step-by-step recipe 1. **Health probe.** Either call `check_queue` via MCP or run `submit_workflow.py --probe`. Stop and report if ComfyUI is unreachable; do not invent a successful run. 2. **Resolve the workflow.** If the user named a file, read it. If they named a workflow by description (e.g. "the one with the upscale at the end"), ask for a path or a saved name. 3. **Apply prompt override if used.** If `--prompt` is provided and the workflow contains `__PROMPT__`, substitute. Otherwise pass the workflow through unchanged. 4. **Submit.** POST to `/prompt` with `{"prompt": <workflow>}`. Capture `prompt_id`. 5. **Poll.** Loop `GET /history/<prompt_id>` with a 2-second sleep. Stop on terminal status (`success`, `error`, or `cancelled`). Use a 15-minute cap; long jobs should be split. 6. **Resolve outputs.** The `outputs` object lists the `SaveImage` / `VHS_VideoCombine` files produced. For each, fetch `GET /view?filename=...&subfolder=...&type=output`. 7. **Report.** Tell the user the prompt_id, the time taken, the saved file paths, and any warnings (low VRAM, retries, partial outputs). ## Long jobs and VRAM safety - ComfyUI 22B-class models can hold the GPU for hours. The Plugin does not implement a watchdog; for jobs longer than 30 minutes, instruct the user to watch `nvidia-smi` themselves. - If a poll returns an error status, capture the full `outputs` block and the error message; report both. Do not retry automatically — generation is expensive. ## Requirements - ComfyUI running locally (or reachable at `COMFYUI_URL`). - Python 3.10+ if using the script. The MCP server requires Node 18+. - No additional accounts, paid services, or network destinations. ## License Apache-2.0. See [LICENSE](../../LICENSE).