comfyui

# ComfyUI Generate images, video, audio, and 3D content through ComfyUI using the official `comfy-cli` for setup/lifecycle and direct REST/WebSocket API for workflow execution. ## What's in this skill **Reference docs (`references/`):** - `official-cli.md` — every `comfy ...` command, with flags - `rest-api.md` — REST + WebSocket endpoints (local + cloud), payload schemas - `workflow-format.md` — API-format JSON, common node types, param mapping **Scripts (`scripts/`):** | Script | Purpose | |--------|---------| | `_common.py` | Shared HTTP, cloud routing, node catalogs (don't run directly) | | `hardware_check.py` | Probe GPU/VRAM/disk → recommend local vs Comfy Cloud | | `comfyui_setup.sh` | Hardware check + comfy-cli + ComfyUI install + launch + verify | | `extract_schema.py` | Read a workflow → list controllable params + model deps | | `check_deps.py` | Check workflow against running server → list missing nodes/models | | `auto_fix_deps.py` | Run check_deps then `comfy node install` / `comfy model download` | | `run_workflow.py` | Inject params, submit, monitor, download outputs (HTTP or WS) | | `run_batch.py` | Submit a workflow N times with sweeps, parallel up to your tier | | `ws_monitor.py` | Real-time WebSocket viewer for executing jobs (live progress) | | `health_check.py` | Verification checklist runner — comfy-cli + server + models + smoke test | | `fetch_logs.py` | Pull traceback / status messages for a given prompt_id | **Example workflows (`workflows/`):** SD 1.5, SDXL, Flux Dev, SDXL img2img, SDXL inpaint, ESRGAN upscale, AnimateDiff video, Wan T2V. See `workflows/README.md`. ## When to Use - User asks to generate images with Stable Diffusion, SDXL, Flux, SD3, etc. - User wants to run a specific ComfyUI workflow file - User wants to chain generative steps (txt2img → upscale → face restore) - User needs ControlNet, inpainting, img2img, or other advanced pipelines - User asks to manage ComfyUI queue, check models, or install custom nodes - User wants video/audio/3D generation via AnimateDiff, Hunyuan, Wan, AudioCraft, etc. ## Architecture: Two Layers ``` ┌─────────────────────────────────────────────────────┐ │ Layer 1: comfy-cli (official lifecycle tool) │ │ Setup, server lifecycle, custom nodes, models │ │ → comfy install / launch / stop / node / model │ └─────────────────────────┬───────────────────────────┘ │ ┌─────────────────────────▼───────────────────────────┐ │ Layer 2: REST/WebSocket API + skill scripts │ │ Workflow execution, param injection, monitoring │ │ POST /api/prompt, GET /api/view, WS /ws │ │ → run_workflow.py, run_batch.py, ws_monitor.py │ └─────────────────────────────────────────────────────┘ ``` **Why two layers?** The official CLI is excellent for installation and server management but has minimal workflow execution support. The REST/WS API fills that gap — the scripts handle param injection, execution monitoring, and output download that the CLI doesn't do. ## Quick Start ### Detect environment ```bash # What's available? command -v comfy >/dev/null 2>&1 && echo "comfy-cli: installed" curl -s http://127.0.0.1:8188/system_stats 2>/dev/null && echo "server: running" # Can this machine run ComfyUI locally? (GPU/VRAM/disk check) python3 scripts/hardware_check.py ``` If nothing is installed, see **Setup & Onboarding** below — but always run the hardware check first. ### One-line health check ```bash python3 scripts/health_check.py # → JSON: comfy_cli on PATH? server reachable? at least one checkpoint? smoke-test passes? ``` ## Core Workflow ### Step 1: Get a workflow JSON in API format Workflows must be in API format (each node has `class_type`). They come from: - ComfyUI web UI → **Workflow → Export (API)** (newer UI) or the legacy "Save (API Format)" button (older UI) - This skill's `workflows/` directory (ready-to-run examples) - Community downloads (civitai, Reddit, Discord) — usually editor format, must be loaded into ComfyUI then re-exported Editor format (top-level `nodes` and `links` arrays) is **not directly executable**. The scripts detect this and tell you to re-export. ### Step 2: See what's controllable ```bash python3 scripts/extract_schema.py workflow_api.json --summary-only # → {"parameter_count": 12, "has_negative_prompt": true, "has_seed": true, ...} python3 scripts/extract_schema.py workflow_api.json # → full schema with parameters, model deps, embedding refs ``` ### Step 3: Run with parameters ```bash # Local (defaults to http://127.0.0.1:8188) python3 scripts/run_workflow.py \ --workflow workflow_api.json \ --args '{"prompt": "a beautiful sunset over mountains", "seed": -1, "steps": 30}' \ --output-dir ./outputs # Cloud (export API key once; uses correct /api routing automatically) export COMFY_CLOUD_API_KEY="comfyui-..." python3 scripts/run_workflow.py \ --workflow workflow_api.json \ --args '{"prompt": "..."}' \ --host https://cloud.comfy.org \ --output-dir ./outputs # Real-time progress via WebSocket (requires `pip install websocket-client`) python3 scripts/run_workflow.py \ --workflow flux_dev.json \ --args '{"prompt": "..."}' \ --ws # img2img / inpaint: pass --input-image to upload + reference automatically python3 scripts/run_workflow.py \ --workflow sdxl_img2img.json \ --input-image image=./photo.png \ --args '{"prompt": "make it watercolor", "denoise": 0.6}' # Batch / sweep: 8 random seeds, parallel up to cloud tier limit python3 scripts/run_batch.py \ --workflow sdxl.json \ --args '{"prompt": "abstract"}' \ --count 8 --randomize-seed --parallel 3 \ --output-dir ./outputs/batch ``` `-1` for `seed` (or omitting it with `--randomize-seed`) generates a fresh random seed per run. ### Step 4: Present results The scripts emit JSON to stdout describing every output file: ```json { "status": "success", "prompt_id": "abc-123", "outputs": [ {"file": "./outputs/sdxl_00001_.png", "node_id": "9", "type": "image", "filename": "sdxl_00001_.png"} ] } ``` ## Decision Tree | User says | Tool | Command | |-----------|------|---------| | **Lifecycle (use comfy-cli)** | | | | "install ComfyUI" | comfy-cli | `bash scripts/comfyui_setup.sh` | | "start ComfyUI" | comfy-cli | `comfy launch --background` | | "stop ComfyUI" | comfy-cli | `comfy stop` | | "install X node" | comfy-cli | `comfy node install <name>` | | "download X model" | comfy-cli | `comfy model download --url <url> --relative-path models/checkpoints` | | "list installed models" | comfy-cli | `comfy model list` | | "list installed nodes" | comfy-cli | `comfy node show installed` | | **Execution (use scripts)** | | | | "is everything ready?" | script | `health_check.py` (optionally with `--workflow X --smoke-test`) | | "what can I change in this workflow?" | script | `extract_schema.py W.json` | | "check if W's deps are met" | script | `check_deps.py W.json` | | "fix missing deps" | script | `auto_fix_deps.py W.json` | | "generate an image" | script | `run_workflow.py --workflow W --args '{...}'` | | "use this image" (img2img) | script | `run_workflow.py --input-image image=./x.png ...` | | "8 variations with random seeds" | script | `run_batch.py --count 8 --randomize-seed ...` | | "show me live progress" | script | `ws_monitor.py --prompt-id <id>` | | "fetch the error from job X" | script | `fetch_logs.py <prompt_id>` | | **Direct REST** | | | | "what's in the queue?" | REST | `curl http://HOST:8188/queue` (local) or `--host https://cloud.comfy.org` | | "cancel that" | REST | `curl -X POST http://HOST:8188/interrupt` | | "free GPU memory" | REST | `curl -X POST http://HOST:8188/free` | ## Setup & Onboarding When a user asks to set up ComfyUI, **the FIRST thing to do is ask whether they want Comfy Cloud (hosted, zero install, API key) or Local (install ComfyUI on their machine)**. Don't start running install commands or hardware checks until they've answered. **Official docs:** https://docs.comfy.org/installation **CLI docs:** https://docs.comfy.org/comfy-cli/getting-started **Cloud docs:** https://docs.comfy.org/get_started/cloud **Cloud API:** https://docs.comfy.org/development/cloud/overview ### Step 0: Ask Local vs Cloud (ALWAYS FIRST) Suggested script: > "Do you want to run ComfyUI locally on your machine, or use Comfy Cloud? > > - **Comfy Cloud** — hosted on RTX 6000 Pro GPUs, all common models pre-installed, > zero setup. Requires an API key (paid subscription required to actually run > workflows; free tier is read-only). Best if you don't have a capable GPU. > - **Local** — free, but your machine MUST meet the hardware requirements: > - NVIDIA GPU with **≥6 GB VRAM** (≥8 GB for SDXL, ≥12 GB for Flux/video), OR > - AMD GPU with ROCm support (Linux), OR > - Apple Silicon Mac (M1+) with **≥16 GB unified memory** (≥32 GB recommended). > - Intel Macs and machines with no GPU will NOT work — use Cloud instead. > > Which would you like?" Routing: - **Cloud** → skip to **Path A**. - **Local** → run hardware check first, then pick a path from Paths B–E based on the verdict. - **Unsure** → run the hardware check and let the verdict decide. ### Step 1: Verify Hardware (ONLY if user chose local) ```bash python3 scripts/hardware_check.py --json # Optional: also probe `torch` for actual CUDA/MPS: python3 scripts/hardware_check.py --json --check-pytorch ``` | Verdict | Meaning | Action | |------------|---------------------------------------------------------------|--------| | `ok` | ≥8 GB VRAM (discrete) OR ≥32 GB unified (Apple Silicon) | Local install — use `comfy_cli_flag` from report | | `marginal` | SD1.5 works; SDXL tight; Flux/video unlikely | Local OK for light workflows, else **Path A (Cloud)** | | `cloud` | No usable GPU, <6 GB VRAM, <16 GB Apple unified, Intel Mac, Rosetta Python | **Switch to Cloud** unless user explicitly forces local | The script also surfaces `wsl: true` (WSL2 with NVIDIA passthrough) and `rosetta: true` (x86_64 Python on Apple Silicon — must reinstall as ARM64). If verdict is `cloud` but the user wants local, do not proceed silently. Show the `notes` array verbatim and ask whether they want to (a) switch to Cloud or (b) force a local install (will OOM or be unusably slow on modern models). ### Choosing an Installation Path Use the hardware check first. The table below is the fallback for when the user has already told you their hardware: | Situation | Recommended Path | |-----------|------------------| | `verdict: cloud` from hardware check | **Path A: Comfy Cloud** | | No GPU / want to try without commitment | **Path A: Comfy Cloud** | | Windows + NVIDIA + non-technical | **Path B: ComfyUI Desktop** | | Windows + NVIDIA + technical | **Path C: Portable** or **Path D: comfy-cli** | | Linux + any GPU | **Path D: comfy-cli** (easiest) | | macOS + Apple Silicon | **Path B: Desktop** or **Path D: comfy-cli** | | Headless / server / CI / agents | **Path D: comfy-cli** | For the fully automated path (hardware check → install → launch → verify): ```bash bash scripts/comfyui_setup.sh # Or with overrides: bash scripts/comfyui_setup.sh --m-series --port=8190 --workspace=/data/comfy ``` It runs `hardware_check.py` internally, refuses to install locally when the verdict is `cloud` (unless `--force-cloud-override`), picks the right `comfy-cli` flag, and prefers `pipx`/`uvx` over global `pip` to avoid polluting system Python. --- ### Path A: Comfy Cloud (No Local Install) For users without a capable GPU or who want zero setup. Hosted on RTX 6000 Pro. **Docs:** https://docs.comfy.org/get_started/cloud 1. Sign up at https://comfy.org/cloud 2. Generate an API key at https://platform.comfy.org/login 3. Set the key: