Runs HuggingFace Diffusers pipelines for Stable Diffusion/SDXL/Flux: txt2img, img2img, inpainting, ControlNet, and LoRA adapters on local GPU. Use when generating or transforming images from Python Diffusers code. Not for ComfyUI node graphs (comfyui), Gradio LoRA Spaces (huggingface-lora-space-builder), or hosted DALL-E/Midjourney APIs.
Runs HuggingFace Diffusers pipelines for Stable Diffusion/SDXL/Flux: txt2img, img2img, inpainting, ControlNet, and LoRA adapters on local GPU. Use when generating or transforming images from Python Diffusers code. Not for ComfyUI node graphs (comfyui), Gradio LoRA Spaces (huggingface-lora-space-builder), or hosted DALL-E/Midjourney APIs.
Comprehensive guide to generating and transforming images with Stable Diffusion using the HuggingFace Diffusers library. Covers text-to-image, image-to-image, inpainting, ControlNet, LoRA adapters, and memory optimization.
When to Use
Use this skill when you need to:
Generate images from natural-language text prompts (text-to-image)
Transform existing images with text guidance (image-to-image / style transfer)
Fill masked regions of an image (inpainting)
Apply spatial conditioning such as edges, poses, or depth maps (ControlNet)
Load and blend LoRA style/character adapters
Build custom diffusion pipelines or batch-generation workflows
import torch
generator = torch.Generator(device="cuda").manual_seed(42)
image = pipe(
prompt="A cat wearing a top hat",
generator=generator,
num_inference_steps=50
).images[0]
5. Image-to-image
from diffusers import AutoPipelineForImage2Image
from PIL import Image
import torch
pipe = AutoPipelineForImage2Image.from_pretrained(
"stable-diffusion-v1-5/stable-diffusion-v1-5",
torch_dtype=torch.float16
).to("cuda")
init_image = Image.open("input.jpg").resize((512, 512))
image = pipe(
prompt="A watercolor painting of the scene",
image=init_image,
strength=0.75, # 0 = no change, 1 = full reimagining
num_inference_steps=50
).images[0]
image.save("output_img2img.png")
6. Inpainting
from diffusers import AutoPipelineForInpainting
from PIL import Image
import torch
pipe = AutoPipelineForInpainting.from_pretrained(
"runwayml/stable-diffusion-inpainting",
torch_dtype=torch.float16
).to("cuda")
image = Image.open("photo.jpg")
mask = Image.open("mask.png") # White = inpaint region, Black = keep
result = pipe(
prompt="A red car parked on the street",
image=image,
mask_image=mask,
num_inference_steps=50
).images[0]
result.save("output_inpaint.png")
7. ControlNet (spatial conditioning)
from diffusers import StableDiffusionControlNetPipeline, ControlNetModel
import torch
controlnet = ControlNetModel.from_pretrained(
"lllyasviel/control_v11p_sd15_canny",
torch_dtype=torch.float16
)
pipe = StableDiffusionControlNetPipeline.from_pretrained(
"stable-diffusion-v1-5/stable-diffusion-v1-5",
controlnet=controlnet,
torch_dtype=torch.float16
).to("cuda")
# control_image = a Canny edge map, OpenPose skeleton, depth map, etc.
image = pipe(
prompt="A beautiful house in the style of Van Gogh",
image=control_image,
num_inference_steps=30
).images[0]
Available ControlNets:
ControlNet
Input Type
Use Case
canny
Edge maps
Preserve structure
openpose
Pose skeletons
Human poses
depth
Depth maps
3D-aware generation
normal
Normal maps
Surface details
mlsd
Line segments
Architectural lines
scribble
Rough sketches
Sketch-to-image
8. LoRA adapters
from diffusers import DiffusionPipeline
import torch
pipe = DiffusionPipeline.from_pretrained(
"stable-diffusion-v1-5/stable-diffusion-v1-5",
torch_dtype=torch.float16
).to("cuda")
# Load single LoRA
pipe.load_lora_weights("path/to/lora", weight_name="style.safetensors")
image = pipe("A portrait in the trained style").images[0]
# Adjust LoRA strength
pipe.fuse_lora(lora_scale=0.8)
# Unload when done
pipe.unload_lora_weights()
# Multiple prompts in one call
prompts = [
"A cat playing piano",
"A dog reading a book",
"A bird painting a picture"
]
images = pipe(prompts, num_inference_steps=30).images
# Multiple images per prompt
images = pipe(
"A beautiful sunset",
num_images_per_prompt=4,
num_inference_steps=30
).images
11. Memory optimization
Apply these in order of increasing aggressiveness:
# 1. Model CPU offload — moves models to CPU when not in use (minimal speed impact)
pipe.enable_model_cpu_offload()
# 2. Sequential CPU offload — more aggressive, slower
pipe.enable_sequential_cpu_offload()
# 3. Attention slicing — compute attention in chunks
pipe.enable_attention_slicing()
# Or: pipe.enable_attention_slicing("max")# 4. xFormers memory-efficient attention (requires xformers package)
pipe.enable_xformers_memory_efficient_attention()
# 5. VAE slicing/tiling for large images
pipe.enable_vae_slicing()
pipe.enable_vae_tiling()
CUDA out of memory: Always call pipe.enable_model_cpu_offload() before pipe.to("cuda") is redundant — use one or the other. If using enable_model_cpu_offload(), do NOT also call .to("cuda"). The offload hook manages device placement.
Black or noise images: Often caused by dtype mismatch. Ensure the entire pipeline uses torch.float16 consistently. If the safety checker destroys valid images, set pipe.safety_checker = None (only for trusted workflows).
Dimensions not multiples of 8: The VAE requires height and width to be multiples of 8 (ideally 64 for SDXL). Non-conforming sizes cause runtime errors or corrupted latents.
SDXL variant="fp16" mismatch: If the model repo does not have an fp16 variant, passing variant="fp16" will fail. Check the model card or omit the argument.
LoRA scale too high: lora_scale above 1.0 can produce burnt or oversaturated images. Keep between 0.5–1.0 for most adapters.
Inpainting mask format: Mask must be a PIL Image where white (255) = region to inpaint and black (0) = region to keep. Grayscale or RGB both work, but ensure the resolution matches the input image.
ControlNet image size: The control image must match the pipeline's target resolution. Resize before passing.
LCM requires LCM scheduler: Loading LCM LoRA without swapping to LCMScheduler produces poor results. Always pair them.
enable_sequential_cpu_offload() is slow: It moves individual layers to GPU one at a time. Use only when enable_model_cpu_offload() is insufficient.
Apple Silicon (MPS): Some operations are not yet supported on MPS. If you hit errors, fall back to CPU: pipe.to("cpu").
Never delete model caches under ~/.cache/huggingface/hub/ (or C:\Users\<you>\.cache\huggingface\hub\) without confirming — re-downloading multi-GB models is costly.
