| name | genai-evaluation-metrics |
| description | Use when evaluating generative models — choosing metrics (FID, IS, KID, sFID, FDD, FVD, PRDC, LPIPS, SSIM, AuthPct, Vendi), setting up online or offline evaluation, feature extractor selection, distributed computation, memory management during sampling. Triggers: "FID", "IS", "KID", "inception score", "frechet", "LPIPS", "SSIM", "evaluation metrics", "generative evaluation", "FVD" |
GenAI Evaluation Metrics
When to Use
- Setting up online evaluation during generative model training
- Choosing which metrics to compute (image vs video, distribution vs perceptual)
- Selecting feature extractors (InceptionV3 vs DINOv2 vs CLIP)
- Configuring sample counts for training-time vs final benchmarks
- Debugging OOM during evaluation sampling phases
- Distributed metric computation with HuggingFace Accelerate
- Evaluating memorization, diversity, or mode collapse
Metric Catalog
Distribution Metrics (Frechet Distance family)
All compute distance between real and generated feature distributions using mean + covariance.
| Metric | Feature Extractor | Dim | What It Captures |
|---|
| FID | InceptionV3 pool3 | 2048 | Overall distribution quality (standard benchmark) |
| sFID | InceptionV3 spatial (Mixed_6e) | 2023 | Spatial structure quality |
| FDD | DINOv2 ViT-L/14 | 1024 | Modern FID alternative, better on textures |
| FVD | I3D | 400 | Video temporal + spatial quality |
Diversity & Quality Metrics
| Metric | What It Measures |
|---|
| IS (Inception Score) | Quality (confident predictions) + diversity (class coverage) |
| KID (Kernel Inception Distance) | Unbiased FID alternative using MMD with polynomial kernel |
| PRDC (Precision/Recall/Density/Coverage) | Manifold overlap: fidelity (P), mode coverage (R), sample density (D), support coverage (C) |
| Vendi Score | Diversity via eigenvalue entropy of similarity matrix |
Perceptual Metrics (paired, per-sample)
| Metric | Feature Extractor | Use Case |
|---|
| LPIPS | AlexNet (spatial) | Perceptual similarity between paired images/frames |
| SSIM | Gaussian filter | Structural similarity (luminance, contrast, structure) |
| PSNR | None (MSE) | Pixel-level reconstruction quality |
Memorization & Overfitting Metrics
| Metric | What It Detects |
|---|
| AuthPct | % of generated samples that are "authentic" (not memorized) |
| CT Score | Data copying / memorization detection |
| FLS (Frechet Likelihood Score) | KDE-based likelihood, sensitive to overfitting |
| FD-infinity | FID extrapolated to infinite samples (removes sample-size bias) |
Patterns
Metric Orchestrator
Wrap all metrics behind a unified update/compute interface:
class MyMetric:
def __init__(self, device="cuda", choices=["fid"]):
if "fid" in choices:
self._fid = FrechetInceptionDistance(
feature=2048, reset_real_features=True,
normalize=False, sync_on_compute=True,
).to(device)
if "is" in choices:
self._is = InceptionScore().to(device)
if "kid" in choices:
self._kid = KernelInceptionDistance(subset_size=50).to(device)
if "prdc" in choices:
self._prdc = PRDC(nearest_k=5).to(device)
if "sfid" in choices:
self._sfid = sFrechetInceptionDistance().to(device)
if "fdd" in choices:
self._fdd = FrechetDinovDistance().to(device)
if "fvd" in choices:
self._fvd = FrechetVideoDistance()
if "dinov2" in choices:
self._dinov2 = DinoV2_Metric().to(device)
def update_real(self, imgs):
for name in self.choices:
if name != "is":
getattr(self, f"_{name}").update(imgs, real=True)
def update_fake(self, imgs):
for name in self.choices:
getattr(self, f"_{name}").update(imgs, real=False)
def compute(self):
results = {}
for name in self.choices:
results.update(getattr(self, f"_{name}").compute())
return results
Image vs video metric selection:
metric = MyMetric(choices=["fid", "is", "kid", "prdc", "sfid", "fdd", "dinov2"])
metric = MyMetric(choices=["fid", "fvd"], video_frame=16)
Online Evaluation (During Training)
Evaluate periodically using the EMA model (not the training model):
if step % cfg.sample_fid_every == 0 and step > 0:
with torch.no_grad():
torch.cuda.empty_cache()
metric.reset()
for _ in range(n_fid_samples // batch_size):
metric.update_real(next(real_data_iter))
for _ in range(n_fid_batches):
z = torch.randn(batch_size, C, H, W, device=device)
samples = sample_fn(z, ema_model)
if use_latent:
samples = vae.decode(samples / 0.18215).sample
samples = (samples.clamp(-1, 1) * 127.5 + 127.5).to(torch.uint8)
samples = accelerator.gather(samples.contiguous())
metric.update_fake(samples)
del samples, z
torch.cuda.empty_cache()
results = metric.compute()
best_fid = min(results["fid"], best_fid)
Track multiple bests for checkpointing:
best_fid = min(results["fid"], best_fid)
best_fdd = min(results["fdd"], best_fdd)
best_sfid = min(results["sfid"], best_sfid)
best_dinov2_fid = min(results["dinov2_fid"], best_dinov2_fid)
sFID, FDD, FVD: Distribution Metric Implementations
sFID uses InceptionV3 spatial features (Mixed_6e), FDD uses DINOv2 ViT-L/14, FVD uses I3D for video.
See references/distribution-metrics.md for implementation details.
DINOv2 Multi-Metric (FID+KID+IS+PRDC in one pass)
Compute FID, KID, IS, and PRDC in DINOv2 feature space with a single extraction pass.
See references/dinov2-multi-metric.md for implementation.
PRDC, AuthPct, Vendi Score, FD-infinity
Diversity and memorization metrics: k-NN manifold overlap (PRDC), memorization detection (AuthPct), eigenvalue diversity (Vendi), and sample-size debiased FID (FD-infinity).
See references/diversity-memorization-metrics.md for implementations.
Video Perceptual Metrics (FVD, LPIPS, SSIM, PSNR)
Frame-by-frame perceptual metrics for video evaluation using I3D, AlexNet, and structural similarity.
See references/video-perceptual-metrics.md for implementation details.
Feature Extractor Selection Guide
| Extractor | Best For | Dim | Speed |
|---|
| InceptionV3 (pool3) | Standard FID benchmarks, paper comparisons | 2048 | Fast |
| InceptionV3 (spatial) | Spatial structure evaluation (sFID) | 2023 | Fast |
| DINOv2 ViT-L/14 | Modern alternative, better texture sensitivity | 1024 | Medium |
| CLIP ViT-L/14 | Text-conditioned generation, cross-modal | varies | Medium |
| I3D | Video quality (FVD) | 400 | Slow |
| AlexNet (LPIPS) | Perceptual similarity (paired) | spatial | Fast |
Sample Count Strategy
evaluation:
sample_fid_n: 5000
sample_fid_every: 20000
sample_fid_bs: 4
num_fid_samples: 50000
Multi-GPU scaling:
fid_batches = cfg.sample_fid_n // (cfg.sample_fid_bs * accelerator.num_processes)
if accelerator.num_processes >= 32:
cfg.sample_fid_n = min(cfg.sample_fid_n, 1000)
Memory Management
with torch.no_grad():
torch.cuda.empty_cache()
metric.reset()
for _ in range(n_batches):
samples = sample_fn(z, ema_model)
samples = accelerator.gather(samples.contiguous())
metric.update_fake(samples)
del samples, z
torch.cuda.empty_cache()
Key rules:
sample_fid_bs <= training batch_size (or OOM)torch.no_grad() around entire eval blockdel + empty_cache() per batch during sampling.contiguous() before accelerator.gather()
Anti-Patterns
- Only computing FID: Use multiple metrics. FID misses spatial structure (sFID), mode collapse (PRDC recall), memorization (AuthPct).
- FID with 50K samples during training: Use 5K for directional signal, 50K for final benchmarks only.
- Using training model for sampling (not EMA): EMA produces better samples. Always use EMA for evaluation.
- Forgetting
accelerator.gather(): Each GPU only sees local samples, metrics computed on partial data.
normalize=True with uint8 images: torchmetrics expects [0,255] uint8 when normalize=False.- InceptionV3 for everything: Consider DINOv2 (FDD) for modern benchmarks, I3D for video (FVD).
- Ignoring memorization: High-quality samples may be copied from training data. Add AuthPct or CT score.
- FID at >=32 GPUs without workaround: torchmetrics sync bug. Cap samples or verify manually.
See Also
gpu-training-acceleration — Memory management during evaluation samplingwandb-experiment-tracking — Logging evaluation metrics to W&B
