| name | experiment-design-for-sciml |
| description | Use when designing experiments, baselines, ablations, generalization tests, and physical diagnostics for CFD-AI and scientific ML manuscripts. |
| version | 0.4.0 |
| author | CFD-AI Paper Skills maintainers |
| metadata | {"short-description":"Experiment matrix design for CFD-AI/SciML papers"} |
Experiment Design for SciML
Trigger
Use when the user claims accuracy, generalization, robustness, physical consistency, efficiency, or journal readiness for a CFD-AI/SciML method.
Core rule
Every experiment must defend a claim. Every claim without an experiment becomes a TODO or is weakened.
Progressive disclosure
Read the relevant gold note before designing a matrix:
| Claim/task | Read |
|---|
| Super-resolution or reconstruction | references/gold-papers/fukami-2019-super-resolution-jfm.md |
| Spatio-temporal super-resolution | references/gold-papers/fukami-2021-spatiotemporal-super-resolution-jfm.md |
| Neural ROM / advection-dominated systems | references/gold-papers/maulik-2021-rom-advection-poF.md |
| Cylinder wake prediction | references/gold-papers/lee-2019-cylinder-wake-jfm.md |
| CNN wake mechanism analysis | references/gold-papers/lee-2021-cnn-wake-analysis-pof.md |
| DRL/control | references/gold-papers/vinuesa-2023-drl-drag-reduction-epje.md |
Workflow
1. List claims
Extract claims:
- accuracy,
- generalization,
- physical consistency,
- efficiency,
- robustness,
- interpretability,
- data efficiency,
- stability.
2. Build baseline matrix
Choose baselines by task:
- surrogate/operator: FNO, DeepONet, U-Net/CNN, GNO/MeshGraphNet where relevant,
- PINN: vanilla PINN, improved PINN variants, numerical solver reference,
- reconstruction: interpolation/POD/autoencoder/super-resolution baselines; for spatio-temporal SR also test temporal interpolation/frame interval,
- closure/ROM: classical ROM, eddy-viscosity or existing closure model,
- control: uncontrolled flow, classical controller, and learned controller under the same simulator budget,
- generative flow: diffusion/FM baseline, deterministic predictor, statistical model.
3. Split design
Require splits that test the actual claim:
- random split = interpolation only,
- temporal split = rollout/generalization,
- parameter split = unseen Re/Ra/Ma/etc.,
- geometry split = unseen shapes,
- boundary-condition split,
- mesh/resolution split,
- OOD split.
4. Ablations
Ablate:
- physics loss/constraint,
- architecture modules,
- spectral/frequency block,
- data size,
- sensor count/sparsity,
- resolution,
- loss weights,
- normalization.
5. Physical diagnostics
Select diagnostics:
- conservation residual,
- BC violation,
- energy spectrum,
- dissipation/enstrophy,
- drag/lift/pressure drop,
- heat flux/Nusselt number,
- rollout stability,
- statistical moments.
5a. v0.4 benchmark-backed checks
| Task | Minimum added evidence |
|---|
| Spatio-temporal SR | spatial downsampling, temporal frame interval, GT/coarse/pred/error, turbulence statistics, and training/frame-interval sensitivity. |
| Wake prediction | Re/regime split, rollout horizon, conservation/force/frequency diagnostics, and named baselines. |
| Neural ROM | POD/Galerkin or classical ROM baseline, latent dimension, rollout stability, and parametric test if claimed. |
| DRL/control | reward, action limits, observations, closed-loop baseline, seeds, and simulator fidelity. |
6. Efficiency reporting
Report:
- training/inference time,
- hardware,
- parameter count,
- memory,
- solver speedup baseline,
- preprocessing cost.
Output template
- Claim list
- Required experiment matrix
- Baselines
- Metrics/physical diagnostics
- Ablations
- Generalization/OOD tests
- Minimum publishable experiment set
- Stretch experiments
Anti-patterns
- Claiming generalization from random snapshot splits.
- Listing baselines without matching data, split, metric, and compute conditions.
- Using RMSE-only evidence for turbulent or physical-consistency claims.
- Omitting failure cases for robustness or OOD claims.
Verification
- Experiments map to claims.
- Baselines are task-appropriate.
- Splits actually test generalization.
- Physical metrics are not replaced by only RMSE.
