bio-clinical-biostatistics-multiplicity-graphical

## Version Compatibility Reference examples tested with: R `gMCP` 0.8.16+, `graphicalMCP` 0.2+, `gatekeeping`, `multcomp`, `multxpert`; Python `statsmodels` 0.14+ for basic FDR/FWER methods. Before using code patterns, verify installed versions match. If versions differ: - R: `packageVersion('<pkg>')` then `?function_name` - Python: `pip show <package>` then `help(module.function)` If code throws ImportError, AttributeError, or TypeError, introspect the installed package and adapt the example to match the actual API rather than retrying. # Multiplicity Control for Confirmatory Trials **"Design the multiplicity strategy for my trial"** -> Specify a closed-testing procedure (graphical, gatekeeping, hierarchical, or step-down Bonferroni-Holm) that controls family-wise error rate at the trial-wide level across primary endpoints, key secondary endpoints, and subgroup analyses, with provable strong FWER control. ## The Foundational Theorem -- Closed Testing Is Necessary **Marcus, Peritz & Gabriel 1976 *Biometrika* 63:655:** a hypothesis H_I (I ⊆ {1,...,m}) is rejected iff every intersection hypothesis ∩_{J⊇I} H_J is rejected by a valid α-level local test. Strong FWER control holds for ANY choice of local tests. **Goeman, Hemerik & Solari 2021 *Ann Stat* 49:1218** tightens this: closed testing is not merely sufficient — it is *necessary* for admissibility under FDP/FWER/k-FWER. **Every admissible multiplicity procedure is equivalent to some closed test.** Graphical procedures, gatekeepers, Hommel, fixed-sequence, fallback — all are closed tests in disguise. **FWER vs FDR philosophical divide:** - **FWER:** P(any false positive among m tests) — regulatory standard for confirmatory inference (agency wants to bound per-trial false-positive rate) - **FDR:** Expected proportion of false discoveries among rejections — exploratory standard (genomics, fMRI, biomarker screens) where many true positives expected Confirmatory clinical trials use FWER essentially universally. ## Algorithmic Taxonomy | Procedure | Type | FWER control | Power profile | Use case | |-----------|------|--------------|---------------|----------| | Bonferroni | Single-step | Yes, any dependence | Conservative; loses 30-50% power vs Hommel under positive dependence | Very small m; worst-case dependence | | Holm 1979 | Step-down | Yes, any dependence | Better than Bonferroni; uniformly dominates | Default for any dependence pattern | | Hochberg 1988 | Step-up | Yes under PRDS (Sarkar 1998) | Better than Holm under PRDS | Positive correlation; verify PRDS | | Hommel 1988 | Step-up via closed tests | Yes under PRDS | Uniformly dominates Hochberg by 1-3% | Whenever Hochberg is valid | | Fixed-sequence (hierarchical) | Sequential | Yes, any dependence | Full alpha for first; subsequent zero if any fail | When clear priority ordering; "key secondary" labelling | | Parallel gatekeeping (Dmitrienko 2003) | Multi-family | Yes | Family-by-family; secondary tested if any primary rejects | Primary family + secondary family | | Serial gatekeeping | Sequential families | Yes | Strict: family k tested only if ALL of family k-1 reject | Co-primary + secondary tiers | | Mixed gatekeeping (Dmitrienko-Tamhane 2008) | Combination | Yes | Combines closed-testing local procedures across families | Complex hierarchies | | Graphical procedures (Bretz-Maurer 2009) | Closed-test as directed graph | Yes by construction | Flexible; allocate alpha to hypotheses via graph weights | Modern standard for confirmatory SAPs | | Graphical + Simes/parametric (Bretz et al 2011) | Closed-test with non-Bonferroni local tests | Yes when Simes valid | Gains power under correlation | Complex co-primary + key secondary + subgroup hierarchies | | Maurer-Bretz 2013 entangled graphs | Memory-augmented graphs | Yes by construction | Alpha propagation depends on origin | Parent-descendant constraints | | Benjamini-Hochberg 1995 | FDR | FDR controlled at level q | Higher power than FWER | Exploratory only; NOT for confirmatory regulatory | **Postdoc reading list:** - Marcus R, Peritz E, Gabriel KR 1976 *Biometrika* 63:655 (closed testing — the foundation) - Goeman JJ, Hemerik J, Solari A 2021 *Ann Stat* 49:1218 (closed testing necessary for admissibility) - Holm S 1979 *Scand J Stat* 6:65 (step-down Bonferroni) - Hochberg Y 1988 *Biometrika* 75:800 (step-up Simes) - Hommel G 1988 *Biometrika* 75:383 (closed Simes; dominates Hochberg) - Sarkar SK 1998/2008 *Ann Stat* (PRDS for Hochberg validity) - Bretz F, Maurer W, Brannath W, Posch M 2009 *Stat Med* 28:586 (graphical procedures — foundational paper) - Bretz F, Posch M, Glimm E, Klinglmueller F, Maurer W, Rohmeyer K 2011 *Biom J* 53:894 (Simes/parametric extensions) - Maurer W, Bretz F 2013 *Stat Med* 32:1739 (entangled graphs / memory) - Dmitrienko A, Offen WW, Westfall PH 2003 *Stat Med* 22:2387 (parallel gatekeeping) - Dmitrienko A, Tamhane AC, Wiens B 2008 *Biom J* (mixed/multistage gatekeeping) - FDA 2022 *Multiple Endpoints in Clinical Trials* Final Guidance (October 2022) - Pocock SJ, Ariti CA, Collier TJ, Wang D 2012 *Eur Heart J* (win-ratio) ## Decision Tree by Scenario | Scenario | Recommended procedure | Why | |----------|----------------------|-----| | 2 co-primary endpoints (both must succeed) | No alpha split needed; per-endpoint alpha-level test; cite FDA 2022 | Co-primary doesn't split alpha; inflates n via joint power | | 2 multiple primary endpoints (any-wins) | Graphical procedure or Holm with weights | Alpha must be allocated; graphical is flexible | | 1 primary + 2 key secondary endpoints | Hierarchical (serial gatekeeping) OR graphical with alpha propagation | Modern SAPs favour graphical | | 1 primary + 3 secondary + 4 subgroup analyses | Graphical procedure via gMCP with pre-specified weights | Complex hierarchies benefit from graph visualisation | | Primary endpoint + tipping-point sensitivity | No multiplicity adjustment needed for sensitivity | Sensitivity is "what if" not "another claim" | | Many exploratory biomarker subgroups | Benjamini-Hochberg FDR | Exploratory; not for label claims | | Win-ratio composite (cardiology) | Single test; no multiplicity | Composite captures multiple events in single hierarchy | | Subgroup analysis (pre-specified) | Graphical alpha allocation; small budget (≤20% by convention); see Dane 2019 for subgroup discipline | Confirmatory subgroup discovery requires explicit allocation | | Adaptive trial with treatment arm dropping | Combination tests (Bauer-Köhne 1994) + closed testing | See clinical-biostatistics/adaptive-designs | | Group-sequential with multiple endpoints | gsDesign or rpact with multivariate alpha spending | Hierarchical alpha across both time and endpoints | ## Bretz-Maurer Graphical Procedures -- The Modern Standard **The Bretz-Maurer-Brannath-Posch 2009 *Stat Med* 28:586 framework recast weighted Bonferroni-Holm closed tests as directed weighted graphs:** - Vertices = elementary null hypotheses with local weights summing to 1 - Directed edges = alpha-propagation rule (when a hypothesis is rejected, its weight redistributes to descendants per edge weights) - The graph IS the procedure: a single visual fully specifies a closed-test procedure across primary, key secondary, and subgroup hierarchies ### gMCP R package ```r library(gMCP) # Construct a graph for primary + 2 key secondary endpoints # Primary endpoint at full alpha; if rejected, alpha propagates equally to secondaries hypotheses <- c('Primary', 'Sec1', 'Sec2') weights <- c(1, 0, 0) # initial alpha all on primary # Transition matrix: rows = source, columns = target # When Primary rejects, weight 0.5 goes to each secondary; when Sec1/Sec2 rejects, alpha returns transitions <- matrix(c( 0, 0.5, 0.5, 0, 0, 1, 0, 1, 0 ), nrow = 3, byrow = TRUE, dimnames = list(hypotheses, hypotheses)) graph <- graphMCP(m = transitions, weights = weights, hnames = hypotheses) # Note: in current gMCP, the graph constructor is `graphMCP(m=, weights=, hnames=)`; # `matrix2graph()` appeared in older tutorials and is not the canonical exported API # -- verify with `?graphMCP` / `?gMCP` in the installed gMCP release before scripting. # Set p-values from the trial p_vals <- c(Primary = 0.018, Sec1 = 0.042, Sec2 = 0.038) # Run the graphical procedure at alpha = 0.025 result <- gMCP(graph, pvalues = p_vals, alpha = 0.025) print(result) # Hierarchical rejection: Primary rejects -> alpha propagates to secondaries -> ... ``` ### Standard SAP graph patterns | Pattern | Graph topology | Use | |---------|----------------|-----| | Pure hierarchical (fixed sequence) | H1 -> H2 -> H3 with weight 1 on each transition | Strict ordering | | Holm graph (equal weights) | Each Hi -> Hj with weight 1/(m-1) | No priority ordering | | Primary + secondaries | Primary -> Sec1 (0.5), Sec2 (0.5); Sec1 ↔ Sec2 (1) | Pivotal labeling claims | | Co-primary chain | H1 -> H2 with full weight if BOTH H1a, H1b reject | Co-primary + secondary | | Subgroup branch | Primary -> Subgroup_OS (0.2), Sec1 (0.4), Sec2 (0.4) | Discovery subgroup with budget | ### Bretz et al 2011 -- Simes and parametric extensions When endpoints are positively correlated, replace the Bonferroni-based intersection test with Simes (for positive dependence) or parametric (using known correlation): ```r library(gMCP) # Use Simes-based local tests at each intersection result_simes <- gMCP(graph, pvalues = p_vals, alpha = 0.025, test = 'Simes') # Or parametric with estimated correlation matrix result_param <- gMCP(graph, pvalues = p_vals, alpha = 0.025, corr = correlation_matrix) ``` ### Maurer-Bretz 2013 entangled graphs **Entangled graphs** add memory: the alpha propagation can depend on the *origin* of the alpha. This allows parent-descendant constraints that a single non-entangled graph cannot express. Example: secondary endpoint Sec1 receives alpha only from Primary, never from Sec2. **Postdoc argument:** purists argue memory makes the procedure non-coherent in Gabriel's sense; Glimm/Maurer/Bretz argue it matches real-world inferential intent. **Gabriel coherence in plain terms:** a coherent procedure rejects a hypothesis H consistently regardless of which superset of H is being tested. Non-entangled graphs are coherent: if H1 is rejected via path A, it would also be rejected via path B. **Entangled (memory-bearing) graphs sacrifice coherence:** the same H may be rejected when alpha arrives from one parent but not from another, because the propagation history changes the available alpha. The trade-off is operational power -- entangled graphs can encode "secondary X is meaningful only if primary Y rejects, not if primary Z rejects" inferential intent that flat coherent procedures cannot express. Choose based on whether the SAP needs path-dependent priority. ## Gatekeeping Procedures ### Serial gatekeeping (hierarchical) Test H1 at full alpha; only if it rejects, test H2 at full alpha; etc. **Maximises power for H1** but H_k becomes inferentially worthless once any H_j (j<k) fails. ```r # Hierarchical / serial: just a chain graph in gMCP hyp <- c('H1', 'H2', 'H3', 'H4') weights <- c(1, 0, 0, 0) trans <- matrix(c(0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0), nrow=4, dimnames=list(hyp, hyp)) graph <- graphMCP(m = trans, weights = weights, hnames = hyp) ``` **Pre-specification of order is critical** — based on clinical importance, NOT expected effect size. Ordering by expected effect is data-driven and inflates Type-I. ### Parallel gatekeeping (Dmitrienko 2003) Secondary family is tested only if *at least one* primary rejects. Bonferroni-based parallel gatekeeper has stepwise representation (Guilbaud 2007 *Biom J* 49:917). ### Mixed / multistage (Dmitrienko-Tamhane 2008) Permits using any closed-testing local procedure (e.g., Holm in family 1, Hommel in family 2) and combining via closure principle. R `gMCP::generalMixGatekeeping` or `Mediana`/`MultXpert` packages. **Postdoc tradeoff:** parallel gatekeeping power loss vs collapsing endpoints into a composite (which avoids multiplicity but dilutes effect if components move in opposite directions); whether tree gatekeeping (Dmitrienko et al 2008 *Stat Med* 27:3446) over-engineers vs equivalent graphical procedure. ## Hochberg vs Hommel vs Holm **Holm 1979** — step-down rejective Bonferroni; FWER controlled under any joint dependence. **Conservative but robust.** **Hochberg 1988** — step-up using ordered Simes critical values; needs Simes inequality which requires PRDS (Sarkar 1998, 2008). Under PRDS, Hochberg uniformly dominates Holm. **Hommel 1988** — also Simes-based but uses closed-testing tableau directly (not step-up shortcut). Uniformly more powerful than Hochberg (typically 1-3% gain). ### When Hochberg fails (anti-conservative) **Hochberg becomes Type-I-inflated under negative dependence** — relevant when comparing endpoints mathematically constrained to move in opposite directions (LDL-C and HDL-C; complementary efficacy and safety endpoints). **Sarkar critique:** when PRDS cannot be proven, fall back to Holm. The lost power is the price of robustness. ```python # Python: statsmodels supports Holm, Hochberg, Hommel from statsmodels.stats.multitest import multipletests p_vals = [0.018, 0.042, 0.038, 0.015] for method in ['holm', 'hochberg', 'hommel', 'bonferroni']: reject, adj_p, _, _ = multipletests(p_vals, alpha=0.05, method=method) print(f'{method}: reject={reject}, adjusted={adj_p}') ``` ## FDA Multiple Endpoints Final Guidance (October 2022) **Federal Register 2022-22882** finalises 2017 draft. Key changes vs draft: - Explicit recognition of newer methods including win-ratio (Pocock 2012 *Eur Heart J*) and weighted composites - Clearer language that "key secondary" endpoints are those for which sponsor wishes to make label claims and which must be in a Type-I-error-controlled hierarchy - Appendix with worked graphical-procedure examples ### Categories | Category | Approach | Note | |----------|----------|------| | Composite | Single test; no multiplicity | Win-ratio, DOOR/RADAR, time-to-first-event | | Co-primary (all-win) | Each at full alpha; n inflated for joint power | Power = product of marginals | | Multiple primary (any-wins) | Alpha must be split (Bonferroni or graphical) | More n required than co-primary if effects similar | | Primary + key secondary | Hierarchical or graphical | Modern preference: graphical for flexibility | **Winner's bias warning:** when post-hoc-selected endpoints are emphasised, bias-corrected effect estimates are recommended (same selection-bias issue as adaptive design). ## The "Almighty Primary Endpoint" Critique **Dmitrienko-D'Agostino 2017 *Stat Med* 36:4423 editorial** surveys progress in trial-multiplicity methodology. A recurring theme motivating that work: insisting on a single primary endpoint can lose power when a therapy has broad multi-domain benefit (heart failure drugs with effects on mortality, hospitalisation, symptoms, biomarkers) -- motivating composite endpoints, the win ratio, or multiple primary endpoints with explicit alpha allocation. **Win-ratio (Pocock-Ariti-Collier-Wang 2012)** and **hierarchical composite (DOOR/RADAR, Evans 2015)** are responses — they preserve a single inferential test while letting multiple endpoints contribute. **FDA counter-position** (Hung, O'Neill, Wang): without a designated primary, sponsors and regulators negotiate over secondary endpoints post hoc, destroying inferential meaning. Hence the FDA 2022 guidance reaffirms key-secondary hierarchies. ## Per-Method Failure Modes ### Hochberg under negative dependence - **Trigger:** Endpoints constrained to move in opposite directions (LDL vs HDL; efficacy vs harm). - **Mechanism:** Hochberg's PRDS assumption fails; Simes inequality doesn't hold; Type-I inflated. - **Symptom:** Replication with Holm finds non-significant where Hochberg rejected. - **Fix:** Switch to Holm (no PRDS assumption); cite Sarkar 1998. ### Fixed-sequence with wrong ordering - **Trigger:** Ordering by expected effect size rather than clinical priority. - **Mechanism:** Data-driven ordering inflates Type-I. - **Symptom:** Reviewer asks for pre-specified ordering rationale. - **Fix:** Pre-specify order by clinical priority in SAP; document rationale. ### Graphical procedure without pre-specified weights - **Trigger:** Weights chosen at analysis time to favour observed results. - **Mechanism:** Equivalent to post-hoc multiplicity tuning; inflates Type-I. - **Symptom:** Multiple "what if" graph variants in CSR. - **Fix:** Pre-specify graph and weights in SAP; document at protocol design. ### Bonferroni when graph would gain power - **Trigger:** Default conservative choice when no thought put into structure. - **Mechanism:** Loses 30-50% power vs Hommel/graphical when m ~ 10 correlated tests. - **Symptom:** Underpowered trial reaches non-significance where graphical procedure would. - **Fix:** Design a proper graph in `gMCP`; cite Bretz-Maurer 2009. ### FDR used for confirmatory primary - **Trigger:** SAP specifies BH-FDR for primary multiplicity. - **Mechanism:** FDR controls expected proportion of false discoveries, not P(any false positive). - **Symptom:** Regulatory reviewer rejects as non-confirmatory. - **Fix:** FWER (graphical, Holm, Hochberg, Hommel) for confirmatory; FDR for exploratory only. ### Subgroup analyses claimed without multiplicity - **Trigger:** Trial reports 10 subgroups with one significant at α=0.05. - **Mechanism:** ~40% probability of at least one false positive under global null. - **Symptom:** Cherry-picked subgroup claim in submission. - **Fix:** Pre-specified graphical alpha allocation OR explicit hypothesis-generating label; cite EMA 2019 subgroup guideline. ### Win-ratio reported without hierarchical priority - **Trigger:** Win-ratio composite with unspecified component priority.