| name | robustness-design |
| description | Design experiments to identify failure boundaries and robustness limits |
| version | 1.0.0 |
| category | experiment-execution |
| type | strategy |
| sops | ["factor-identification","level-specification","baseline-selection","metric-specification","sample-size-estimation","design-matrix-construction"] |
| tactics | ["statistical-method-selection"] |
| dependencies | {"sops":["baseline-selection","design-matrix-construction","factor-identification","level-specification","metric-specification","sample-size-estimation"],"tactics":["statistical-method-selection"]} |
Strategy: Robustness Design
Question: Under what conditions does the method fail?
Methodology
- Distribution Shift Testing: Evaluate under covariate shift, label shift, domain shift.
- Adversarial Robustness: Perturbation-based attacks (PGD, AutoAttack) at varying epsilon.
- Cross-Domain Transfer: Test on domains not seen during training.
- Noise Injection: Gaussian noise, label noise, missing data at varying severity.
- Stress Testing: Push inputs to boundary conditions (extreme lengths, rare categories, edge cases).
Execution Flow
- factor-identification → Identify robustness dimensions (noise type, shift type, severity)
- level-specification → Define severity levels for each perturbation
- baseline-selection → Select robust baselines for comparison
- metric-specification → Define degradation metrics (absolute and relative to clean)
- design-matrix-construction → Build perturbation grid
- sample-size-estimation → Determine samples needed per condition
- statistical-method-selection (tactic) → Choose tests for degradation significance
Budget Gate
| Robustness Type | Conditions | Severities | Min Runs | Notes |
|---|
| Single perturbation | 1 | 3-5 | 3-5 | Quick sanity check |
| Multi-perturbation | 3-5 | 3 each | 9-15 | Standard robustness eval |
| Adversarial sweep | 1 attack | 5-10 epsilon | 5-10 | Adversarial robustness curve |
| Comprehensive | 5+ types | 3-5 each | 50+ | Publication-ready robustness |
| Cross-domain | N domains | 1 | N | Transfer evaluation |
Available Tactics
Optional, no fixed order; the final leaf is always a sop.
