debug-model

# Debug SD.Next And Diffusers Model Port Read the error, identify which integration layer is failing, isolate the smallest reproducible failure point, fix the root cause, and validate the fix without expanding scope. ## When To Use - A newly added SD.Next model type does not autodetect correctly - The loader fails to instantiate a pipeline or component - A custom pipeline imports but fails during `from_pretrained` - Prompt encoding fails because of tokenizer, processor, or text encoder mismatch - Sampling fails due to tensor shape, dtype, device, or scheduler issues - The model loads but outputs corrupted images, wrong output type, or obviously incorrect results ## Guidance - Consult `.github/instructions/core.instructions.md` for relevant core runtime and model debugging guidance before proceeding. ## Debugging Order Always debug from the outside in. 1. Integration entry checks: detection and routing. 2. Load path checks: loader arguments and component selection, checkpoint path and artifact layout, then weight loading and key mapping. 3. Runtime path checks: prompt encoding, sampling forward path, and output postprocessing plus SD.Next task integration. Do not start by rewriting the architecture when the failure appears in detection, loader wiring, or output handling. Only consider architecture rewrites after these layers are validated and the root cause is confirmed to be architectural. ## Files To Check First - `.github/copilot-instructions.md` - `.github/instructions/core.instructions.md` - `modules/sd_detect.py` - `modules/sd_models.py` - `modules/modeldata.py` - `pipelines/model_<name>.py` - `pipelines/<model>/model.py` - `pipelines/<model>/pipeline.py` - `pipelines/generic.py` If the port is based on a standalone script, compare the failing path against the original reference implementation and identify the first semantic divergence. ## Failure Classification ### 1. Model Not Detected Or Misclassified Check: - Filename and repo-name heuristics in `modules/sd_detect.py` - Loader dispatch branch in `modules/sd_models.py` - Reverse pipeline classification in `modules/modeldata.py` Typical symptoms: - Wrong loader called - Pipeline classified as a broader family such as `chroma` instead of a custom `zetachroma` - Task switching behaves incorrectly because the loaded pipeline type is wrong ### 2. Loader Fails Before Pipeline Construction Check: - `sd_models.path_to_repo(checkpoint_info)` output - `generic.load_transformer(...)` and `generic.load_text_encoder(...)` arguments - Duplicate kwargs such as `torch_dtype` - Wrong class chosen for text encoder, tokenizer, or processor - Whether the source is really a Diffusers repo or only a raw checkpoint Typical symptoms: - Missing subfolder errors - `from_pretrained` argument mismatch - Component class mismatch ### 3. Raw Checkpoint Load Fails Check: - Checkpoint path resolution for local file, local directory, and Hub repo - State dict load method - Key remapping logic - Config inference from tensor shapes - Missing versus unexpected keys after `load_state_dict` Typical symptoms: - Key mismatch explosion - Wrong inferred head counts, dimensions, or decoder settings - Silent shape corruption caused by a bad remap ### 4. Prompt Encoding Fails Check: - Tokenizer or processor choice - `trust_remote_code` requirements - Chat template or custom prompt formatting - Hidden state index selection - Padding and batch alignment between positive and negative prompts Typical symptoms: - Tokenizer attribute errors - Hidden state shape mismatch - CFG failures when negative prompts do not match prompt batch length ### 5. Sampling Or Forward Pass Fails Check: - Input tensor shape and channel count - Device and dtype alignment across all components - Scheduler timesteps and expected timestep convention - Classifier-free guidance concatenation and split logic - Pixel-space versus latent-space assumptions Typical symptoms: - Shape mismatch in attention or decoder blocks - Device mismatch between text encoder output and model tensors - Images exploding to NaNs because timestep semantics are inverted ### 6. Output Is Wrong But No Exception Is Raised Check: - Whether the model predicts `x0`, noise, or velocity - Whether the Euler or other sampler update matches the model objective - Final scaling and clamp path - `output_type` handling and `pipe.task_args` - Whether a VAE is being applied incorrectly to direct pixel-space output Typical symptoms: - Black, gray, washed-out, or heavily clipped images - Output with correct size but obviously broken semantics - Correct tensors but wrong SD.Next display behavior because output type is mismatched ## Minimal Debug Procedure ### 1. Reproduce Narrowly Capture the smallest failing operation. - Pure import failure - Loader-only failure - `from_pretrained` failure - Prompt encode failure - Single forward pass failure - First sampler step failure Prefer narrow Python checks before attempting a full generation run. ### 2. Compare Against Working Pattern Find the closest working in-repo analogue and compare: - Loader structure - Registered module names - Pipeline class name and module registration - Prompt encoding path - Output conversion path ### 3. Fix The Root Cause Examples: - Add the missing `modeldata` branch instead of patching downstream task handling - Fix checkpoint remapping rather than forcing `strict=False` and ignoring real mismatches - Correct the output path for pixel-space models instead of routing through a VAE - Make config inference fail explicitly when ambiguous instead of guessing silently ### 4. Validate In Layers After each meaningful fix, validate the narrowest relevant layer first. - `compileall` or syntax check - `ruff` on touched files - Import smoke test - Loader-only smoke test - Full run only when the lower layers are stable ## Common Root Causes - `modules/modeldata.py` not updated after adding a new custom pipeline family - `modules/sd_detect.py` branch order causes overbroad detection to win first - Loader passes duplicated keyword args like `torch_dtype` - Shared text encoder assumptions do not match the actual model variant - `from_pretrained` assumes `transformer/` or `text_encoder/` subfolders that do not exist - Key remapping merges QKV in the wrong order - CFG path concatenates embeddings or latents incorrectly - Direct pixel-space models are postprocessed like latent-space diffusion outputs - Negative prompts are not padded or repeated to match prompt batch shape - Pipeline class naming collides with broader family checks in `modeldata` ## Validation Checklist When closing the task, report which of these were completed: 1. Exact failing layer identified 2. Root cause fixed 3. Syntax check passed 4. Focused lint passed 5. Import or loader smoke test passed 6. Real generation tested, or explicitly not tested ## Example Request Shapes - "The new model port fails in from_pretrained" - "SD.Next detects my custom pipeline as the wrong model type" - "The loader works but generation returns black images" - "This standalone-script port loads weights but crashes in attention"