Runs HuggingFace Diffusers pipelines for Stable Diffusion/SDXL/Flux: txt2img, img2img, inpainting, ControlNet, and LoRA adapters on local GPU. Use when generating or transforming images from Python Diffusers code. Not for ComfyUI node graphs (comfyui), Gradio LoRA Spaces (huggingface-lora-space-builder), or hosted DALL-E/Midjourney APIs.
Standardmäßig ist der Prompt ausgewählt, der zuerst die Quelle prüft. Sie können zu einem direkten Befehl wechseln oder eine lokale Kopie herunterladen.
Quelldateien prüfen
Lesen Sie SKILL.md und alle von SkillsMP angezeigten Begleitdateien, bevor Sie sich für eine Installation entscheiden.
Mit Codex oder Claude installieren Kopieren Sie diesen Prompt, fügen Sie ihn in Codex, Claude oder einen anderen Assistant ein und lassen Sie die Skill-Seite prüfen und installieren.
Ein direkter Befehl überspringt den Prüf-Prompt. Prüfen Sie die Quelle, bevor Sie ihn ausführen.
Runs HuggingFace Diffusers pipelines for Stable Diffusion/SDXL/Flux: txt2img, img2img, inpainting, ControlNet, and LoRA adapters on local GPU. Use when generating or transforming images from Python Diffusers code. Not for ComfyUI node graphs (comfyui), Gradio LoRA Spaces (huggingface-lora-space-builder), or hosted DALL-E/Midjourney APIs.
Comprehensive guide to generating and transforming images with Stable Diffusion using the HuggingFace Diffusers library. Covers text-to-image, image-to-image, inpainting, ControlNet, LoRA adapters, and memory optimization.
When to Use
Use this skill when you need to:
Generate images from natural-language text prompts (text-to-image)
Transform existing images with text guidance (image-to-image / style transfer)
Fill masked regions of an image (inpainting)
Apply spatial conditioning such as edges, poses, or depth maps (ControlNet)
Load and blend LoRA style/character adapters
Build custom diffusion pipelines or batch-generation workflows
import torch
generator = torch.Generator(device="cuda").manual_seed(42)
image = pipe(
prompt="A cat wearing a top hat",
generator=generator,
num_inference_steps=50
).images[0]
5. Image-to-image
from diffusers import AutoPipelineForImage2Image
from PIL import Image
import torch
pipe = AutoPipelineForImage2Image.from_pretrained(
"stable-diffusion-v1-5/stable-diffusion-v1-5",
torch_dtype=torch.float16
).to("cuda")
init_image = Image.open("input.jpg").resize((512, 512))
image = pipe(
prompt="A watercolor painting of the scene",
image=init_image,
strength=0.75, # 0 = no change, 1 = full reimagining
num_inference_steps=50
).images[0]
image.save("output_img2img.png")
6. Inpainting
from diffusers import AutoPipelineForInpainting
from PIL import Image
import torch
pipe = AutoPipelineForInpainting.from_pretrained(
"runwayml/stable-diffusion-inpainting",
torch_dtype=torch.float16
).to("cuda")
image = Image.open("photo.jpg")
mask = Image.open("mask.png") # White = inpaint region, Black = keep
result = pipe(
prompt="A red car parked on the street",
image=image,
mask_image=mask,
num_inference_steps=50
).images[0]
result.save("output_inpaint.png")
7. ControlNet (spatial conditioning)
from diffusers import StableDiffusionControlNetPipeline, ControlNetModel
import torch
controlnet = ControlNetModel.from_pretrained(
"lllyasviel/control_v11p_sd15_canny",
torch_dtype=torch.float16
)
pipe = StableDiffusionControlNetPipeline.from_pretrained(
"stable-diffusion-v1-5/stable-diffusion-v1-5",
controlnet=controlnet,
torch_dtype=torch.float16
).to("cuda")
# control_image = a Canny edge map, OpenPose skeleton, depth map, etc.
image = pipe(
prompt="A beautiful house in the style of Van Gogh",
image=control_image,
num_inference_steps=30
).images[0]
Available ControlNets:
ControlNet
Input Type
Use Case
canny
Edge maps
Preserve structure
openpose
Pose skeletons
Human poses
depth
Depth maps
3D-aware generation
normal
Normal maps
Surface details
mlsd
Line segments
Architectural lines
scribble
Rough sketches
Sketch-to-image
8. LoRA adapters
from diffusers import DiffusionPipeline
import torch
pipe = DiffusionPipeline.from_pretrained(
"stable-diffusion-v1-5/stable-diffusion-v1-5",
torch_dtype=torch.float16
).to("cuda")
# Load single LoRA
pipe.load_lora_weights("path/to/lora", weight_name="style.safetensors")
image = pipe("A portrait in the trained style").images[0]
# Adjust LoRA strength
pipe.fuse_lora(lora_scale=0.8)
# Unload when done
pipe.unload_lora_weights()
# Multiple prompts in one call
prompts = [
"A cat playing piano",
"A dog reading a book",
"A bird painting a picture"
]
images = pipe(prompts, num_inference_steps=30).images
# Multiple images per prompt
images = pipe(
"A beautiful sunset",
num_images_per_prompt=4,
num_inference_steps=30
).images
11. Memory optimization
Apply these in order of increasing aggressiveness:
# 1. Model CPU offload — moves models to CPU when not in use (minimal speed impact)
pipe.enable_model_cpu_offload()
# 2. Sequential CPU offload — more aggressive, slower
pipe.enable_sequential_cpu_offload()
# 3. Attention slicing — compute attention in chunks
pipe.enable_attention_slicing()
# Or: pipe.enable_attention_slicing("max")# 4. xFormers memory-efficient attention (requires xformers package)
pipe.enable_xformers_memory_efficient_attention()
# 5. VAE slicing/tiling for large images
pipe.enable_vae_slicing()
pipe.enable_vae_tiling()
CUDA out of memory: Always call pipe.enable_model_cpu_offload() before pipe.to("cuda") is redundant — use one or the other. If using enable_model_cpu_offload(), do NOT also call .to("cuda"). The offload hook manages device placement.
Black or noise images: Often caused by dtype mismatch. Ensure the entire pipeline uses torch.float16 consistently. If the safety checker destroys valid images, set pipe.safety_checker = None (only for trusted workflows).
Dimensions not multiples of 8: The VAE requires height and width to be multiples of 8 (ideally 64 for SDXL). Non-conforming sizes cause runtime errors or corrupted latents.
SDXL variant="fp16" mismatch: If the model repo does not have an fp16 variant, passing variant="fp16" will fail. Check the model card or omit the argument.
LoRA scale too high: lora_scale above 1.0 can produce burnt or oversaturated images. Keep between 0.5–1.0 for most adapters.
Inpainting mask format: Mask must be a PIL Image where white (255) = region to inpaint and black (0) = region to keep. Grayscale or RGB both work, but ensure the resolution matches the input image.
ControlNet image size: The control image must match the pipeline's target resolution. Resize before passing.
LCM requires LCM scheduler: Loading LCM LoRA without swapping to LCMScheduler produces poor results. Always pair them.
enable_sequential_cpu_offload() is slow: It moves individual layers to GPU one at a time. Use only when enable_model_cpu_offload() is insufficient.
Apple Silicon (MPS): Some operations are not yet supported on MPS. If you hit errors, fall back to CPU: pipe.to("cpu").
Never delete model caches under ~/.cache/huggingface/hub/ (or C:\Users\<you>\.cache\huggingface\hub\) without confirming — re-downloading multi-GB models is costly.
