| name | generative-models |
| description | This skill should be used when the user asks about "generative models", "diffusion models", "DDPM", "DDIM", "stable diffusion", "GANs", "GAN training", "generator", "discriminator", "mode collapse", "FID score", "language models", "LLM", "GPT", "transformer", "text generation", "image generation", "ELBO", "score matching", "latent diffusion", "variational autoencoder", "VAE", "CLIP", "multimodal generation", or when building any system that generates novel data samples. |
| version | 1.0.0 |
Generative Models — Diffusion, GANs, and Transformer LMs
Provides architecture patterns, loss function derivations, training stability techniques, and evaluation protocols for diffusion models, GANs, and large-scale language models.
Diffusion Models
Mathematical Foundation:
- Forward process (noising): q(x_t | x_{t-1}) = N(x_t; √(1−β_t)·x_{t-1}, β_t·I)
- Closed-form marginal: q(x_t | x_0) = N(x_t; √ᾱ_t·x_0, (1−ᾱ_t)·I)
- Reverse process (denoising): learn p_θ(x_{t-1} | x_t) by predicting ε_θ(x_t, t) ≈ ε
Training Objective (simplified ELBO):
L_simple = E_{t, x_0, ε}[||ε − ε_θ(√ᾱ_t·x_0 + √(1−ᾱ_t)·ε, t)||²]
Architecture: U-Net with:
- Residual blocks at each resolution
- Self-attention at low resolutions (32×32 and 16×16)
- Sinusoidal time-step embedding conditioning (d = 256)
- GroupNorm (32 groups) for training stability
Noise Schedules: linear (DDPM), cosine (improved DDPM), or sigmoid (more uniform SNR)
Sampling Acceleration: DDIM (50 steps), DPM-Solver (10–20 steps) vs. DDPM (1000 steps)
See references/diffusion-models.md for full derivation, implementation, and latent diffusion (LDM) patterns.
Generative Adversarial Networks
Minimax Objective:
min_G max_D V(D,G) = E_{xp_data}[log D(x)] + E_{zp_z}[log(1 − D(G(z)))]
Critical Stabilization Techniques (apply all for stable training):
- Spectral normalization on D: constrains Lipschitz constant, prevents discriminator from becoming too powerful
- Label smoothing: real labels = 0.9 (not 1.0), fake labels = 0.1 (not 0.0)
- WGAN-GP gradient penalty: λ·E[(||∇D(x̂)||₂ − 1)²], λ = 10, eliminates mode collapse risk
- TTUR: use LR_D = 4e-4, LR_G = 1e-4 (different timescales stabilize training)
- MiniBatch discrimination: prevents G from always generating the same output
Evaluation Metric: FID (Fréchet Inception Distance) — lower is better; FID < 10 is state-of-the-art for faces.
See references/gans.md for architecture variants (DCGAN → StyleGAN2 → ProjectedGAN) and CTGAN for tabular data.
Transformer Language Models
Core Architecture (decoder-only, GPT-style):
- Token embedding (vocab_size → d_model) + positional encoding (RoPE preferred)
- N × blocks: [LayerNorm → MHA → residual] → [LayerNorm → FFN(4×d) → residual]
- Attention: Attention(Q,K,V) = softmax(QK^T / √d_k)·V
- Output: linear projection to vocab_size → softmax → token probabilities
Scaling Guidance (Chinchilla-optimal, Hoffmann et al. 2022):
- Optimal tokens T for N parameters: T_opt = 20·N
- Equal scaling of model size and data is key — do not over- or under-train
Training Techniques for LLMs:
- Mixed precision BF16 (A100+), FP16 + GradScaler (V100)
- FlashAttention-2 for memory-efficient O(n) attention
- Gradient accumulation to achieve large effective batch sizes without large VRAM
- Learning rate: warmup to peak over 1% of steps, then cosine decay to 10% of peak
See references/transformer-lms.md for full implementation, tokenization, and RLHF fine-tuning patterns.
Quality Evaluation by Model Type
| Model Type | Primary Metric | Secondary Metrics |
|---|
| Diffusion (images) | FID (↓) | IS (↑), CLIP score, human preference |
| GAN (images) | FID (↓) | Precision, Recall, coverage |
| Language Model | Perplexity (↓) | BLEU, ROUGE, BERTScore, task benchmarks |
| Conditional generation | Alignment score | Diversity, fidelity vs. reference |
