stable-diffusion

name	stable-diffusion
description	Run Stable Diffusion locally with diffusers — text-to-image, img2img, inpainting, ControlNet, and SDXL.
version	1.0.0
author	hermes-CCC (ported from Hermes Agent by NousResearch)
license	MIT
metadata	{"hermes":{"tags":["MLOps","Stable-Diffusion","Image-Generation","Diffusers","Text-to-Image"],"related_skills":[]}}

Stable Diffusion

Purpose

• Use this skill to generate or edit images locally with Hugging Face diffusers.
• Prefer it for text-to-image, img2img, inpainting, and model composition workflows.
• This skill covers both classic Stable Diffusion models and SDXL.
• It is useful for scripted generation, reproducible experiments, and GPU-backed image pipelines.

Install

pip install diffusers transformers accelerate torch

• You typically also need a compatible CUDA-enabled PyTorch build for GPU inference.
• Confirm the install in Python before pulling large checkpoints.

Core Libraries

• diffusers for pipeline abstractions
• transformers for text encoders and related model components
• accelerate for efficient device loading and memory movement
• torch for runtime execution

Text-to-Image With SD 1.5

• Classic Stable Diffusion 1.5 uses StableDiffusionPipeline.

import torch
from diffusers import StableDiffusionPipeline

pipe = StableDiffusionPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    torch_dtype=torch.float16,
)
pipe = pipe.to("cuda")

image = pipe(
    prompt="a cinematic photo of a mountain observatory at sunrise",
    negative_prompt="blurry, low quality, distorted",
    num_inference_steps=30,
    guidance_scale=7.5,
).images[0]

image.save("output.png")

SDXL

• SDXL typically uses StableDiffusionXLPipeline.

import torch
from diffusers import StableDiffusionXLPipeline

pipe = StableDiffusionXLPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16,
)
pipe = pipe.to("cuda")

image = pipe(
    prompt="a highly detailed editorial photo of a futuristic library interior",
    negative_prompt="low resolution, deformed, extra limbs",
    num_inference_steps=35,
    guidance_scale=6.5,
).images[0]

image.save("sdxl-output.png")

• SDXL generally produces stronger prompt fidelity and image quality than SD 1.5, but it also requires more VRAM.

Key Parameters

• prompt: the main text instruction
• negative_prompt: what to suppress
• num_inference_steps: denoising step count
• guidance_scale: classifier-free guidance strength
• seed: random seed for reproducibility

Seeded Generation

• Use a seed when you need repeatable outputs:

import torch
from diffusers import StableDiffusionPipeline

pipe = StableDiffusionPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    torch_dtype=torch.float16,
).to("cuda")

generator = torch.Generator(device="cuda").manual_seed(42)

image = pipe(
    prompt="a clean product photo of a ceramic mug on a wood table",
    negative_prompt="blurry, noisy, warped",
    num_inference_steps=28,
    guidance_scale=7.0,
    generator=generator,
).images[0]

image.save("seeded-output.png")

• The same seed and settings help with debugging prompt and LoRA changes.

Save Output

• Save the generated image with PIL:

image.save("output.png")

• Always save prompt metadata separately if you need auditability or experiment tracking.

Img2Img

• Use StableDiffusionImg2ImgPipeline to transform an existing image while preserving composition.

import torch
from diffusers import StableDiffusionImg2ImgPipeline
from PIL import Image

pipe = StableDiffusionImg2ImgPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    torch_dtype=torch.float16,
).to("cuda")

init_image = Image.open("input.png").convert("RGB").resize((768, 768))

image = pipe(
    prompt="turn this concept sketch into a polished sci-fi matte painting",
    negative_prompt="blurry, low contrast, artifacts",
    image=init_image,
    strength=0.65,
    num_inference_steps=30,
    guidance_scale=7.5,
).images[0]

image.save("img2img-output.png")

• Lower strength preserves more of the input image.
• Higher strength pushes the result further away from the source.

Inpainting

• Use StableDiffusionInpaintPipeline to replace or repair masked regions.

import torch
from diffusers import StableDiffusionInpaintPipeline
from PIL import Image

pipe = StableDiffusionInpaintPipeline.from_pretrained(
    "runwayml/stable-diffusion-inpainting",
    torch_dtype=torch.float16,
).to("cuda")

image = Image.open("scene.png").convert("RGB").resize((512, 512))
mask = Image.open("mask.png").convert("RGB").resize((512, 512))

result = pipe(
    prompt="replace the missing area with a wooden chair",
    negative_prompt="blurry, malformed, duplicate objects",
    image=image,
    mask_image=mask,
    num_inference_steps=30,
    guidance_scale=7.5,
).images[0]

result.save("inpaint-output.png")

• White mask regions are typically where edits are applied.
• Good masks matter as much as prompts for reliable inpainting.

ControlNet

• ControlNet is useful when you want stronger control over pose, depth, edges, or composition.
• Typical uses include pose-guided character generation, depth-aware edits, and line-art conditioning.
• Pair ControlNet with SD 1.5 or SDXL depending on the model combination you are using.

Memory Optimization

• Reduce memory pressure with built-in helpers:

pipe.enable_model_cpu_offload()
pipe.enable_attention_slicing()

• pipe.enable_model_cpu_offload() is often helpful on constrained GPUs.
• pipe.enable_attention_slicing() can reduce peak memory at some performance cost.
• These settings are practical for laptops and single-GPU consumer machines.

LoRA Loading

• Load LoRA adapters to specialize style, subject, or composition behavior:

pipe.load_lora_weights("./lora.safetensors")

• Keep the base model and LoRA pairing compatible.
• Track LoRA names, weights, and prompts in experiment logs.

Negative Prompts

• Common negative prompts include:

• blurry

• low quality

• worst quality

• deformed

• extra limbs

• bad anatomy

• artifact

• text

• watermark

• Use concise negative prompts first.

• Overly long negative prompts can produce unstable or muddled outputs.

Prompting Guidance

• Be concrete about subject, style, lighting, framing, and medium.
• Use short prompt iterations during tuning rather than changing many variables at once.
• Record prompt, negative prompt, seed, and model version together.

SDXL vs SD 1.5

• Use SDXL when:

• prompt fidelity matters

• you need stronger detail and composition

• you have enough VRAM

• Use SD 1.5 when:

• you need a lighter model

• you rely on mature community tooling

• you need broad LoRA and ControlNet ecosystem support

Common Workflows

• Text-to-image concept generation
• Product mockups and ideation
• Img2img refinement from sketches
• Inpainting object replacement
• Style transfer through LoRAs

ComfyUI Alternative

• diffusers is strong for code-driven workflows.
• ComfyUI is a strong GUI alternative when you want node-based visual workflows.
• Use ComfyUI for rapid experimentation, complex graph composition, or collaborative prompt workflows.

Practical GPU Guidance

• SD 1.5 is easier on smaller GPUs.
• SDXL generally needs more VRAM and benefits from float16.
• CPU generation is possible, but it is much slower and rarely ideal for interactive use.

Common Failure Modes

• Out-of-memory:

• enable CPU offload

• enable attention slicing

• reduce image size

• use SD 1.5 instead of SDXL

• Muddy or low-quality images:

• increase num_inference_steps

• refine the prompt

• simplify the negative prompt

• verify you are using the intended model

• Unreliable edits in img2img:

• lower or raise strength depending on whether the source is being ignored or over-preserved

• use clearer prompts

• start from a cleaner input image

• Inpainting artifacts:

• improve the mask

• widen the masked area slightly

• use a prompt that matches the surrounding scene

Recommended Workflow

• Start with a baseline text-to-image run.
• Lock a seed when comparing prompt or LoRA changes.
• Move to img2img or inpainting only after the base model behavior looks correct.
• Add memory optimizations before assuming you need larger hardware.

When To Use This Skill

• You need local image generation from Python.
• You want reproducible scripted generation for experiments or pipelines.
• You need SDXL, img2img, inpainting, or LoRA-based customization.
• You prefer code-first workflows over GUI-only tools.

Quick Reference

• Install: pip install diffusers transformers accelerate torch
• SDXL pipeline: StableDiffusionXLPipeline.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0")
• SD 1.5 pipeline: StableDiffusionPipeline
• Save image: image.save("output.png")
• Img2img: StableDiffusionImg2ImgPipeline
• Inpainting: StableDiffusionInpaintPipeline
• Memory helpers: pipe.enable_model_cpu_offload() and pipe.enable_attention_slicing()
• LoRA: pipe.load_lora_weights("./lora.safetensors")