| name | bio-experimental-design-multiple-testing |
| description | Controls error rates across thousands of simultaneous tests in genomics discovery using false-discovery-rate methods (Benjamini-Hochberg 1995; Benjamini-Yekutieli 2001 for arbitrary dependence; Storey q-value with pi0 estimation; local FDR; independent filtering Bourgon 2010; covariate-weighted FDR via IHW Ignatiadis 2016), plus family-wise error control (Bonferroni, Holm) and the GWAS genome-wide threshold. Covers the FDR-versus-FWER choice as the discovery-versus-confirmatory distinction, the dependence assumptions behind BH (PRDS) versus BY, pi0 estimation, the independent-filtering and false-coverage-rate traps, and reproducibility ranking via IDR (Li 2011). Use when correcting p-values from genome-wide tests, choosing between BH/BY/q-value/Bonferroni, setting an FDR threshold, applying IHW or independent filtering, or interpreting q-values. For confirmatory trials with few pre-specified endpoints (closed testing, graphical/gatekeeping), see clinical-biostatistics/multiplicity-graphical. |
| goal_approach_exempt | true |
| origin | openai4s |
| category | bioskills/experimental-design |
| metadata | {"tool_type":"mixed","primary_tool":"qvalue","third_party":{"name":"GPTomics/bioSkills","repository":"https://github.com/GPTomics/bioSkills","commit":"d91ed3d563019e649dc854c56ccd62551359488a","license":"MIT"}} |
Version Compatibility
Reference examples tested with: qvalue 2.34+, IHW 1.30+, R stats (base) p.adjust, statsmodels 0.14+, scipy 1.12+.
Before using code patterns, verify installed versions match. If versions differ:
- R:
packageVersion('<pkg>') then ?function_name to verify parameters
- Python:
pip show <package> then help(module.function) to check signatures
If code throws an error, introspect the installed package and adapt to the actual API. Note: statsmodels.stats.multitest.multipletests defaults to method='hs' (Holm-Sidak, an FWER method), NOT Benjamini-Hochberg — always pass method='fdr_bh'/'fdr_by'/'bonferroni'/'holm' explicitly.
Multiple Testing Correction
"Correct p-values for testing thousands of features" -> Choose an error rate appropriate to the regime (FDR for discovery, FWER for confirmatory), apply a procedure whose dependence assumptions match the data, and report the adjusted quantity with its interpretation.
- R:
p.adjust(p, method = 'BH'), qvalue::qvalue(), IHW::ihw()
- Python:
statsmodels.stats.multitest.multipletests(p, method='fdr_bh')
The Single Most Important Modern Insight -- FDR vs FWER Is a Choice About Which Error Matters
The choice between false-discovery-rate and family-wise-error control is not a technicality; it is a statement about which kind of mistake is costly. In discovery (20,000 genes, thousands of peaks), tolerating a small, controlled fraction of false positives among the rejections buys enormous power — FDR is the right currency, and Bonferroni would discard nearly every true effect. In confirmatory work (a handful of pre-specified endpoints), a single false positive is unacceptable and FWER/closed testing is the standard (that regime lives in clinical-biostatistics/multiplicity-graphical). Two further levers buy back power that plain BH leaves on the table: estimating pi0 (the proportion of true nulls) turns BH into the more powerful q-value (Storey 2002 J R Stat Soc B 64:479; Storey & Tibshirani 2003 PNAS 100:9440), and weighting hypotheses by an independent informative covariate recovers power via IHW (Ignatiadis 2016 Nat Methods 13:577). The dependence structure matters: BH controls FDR under independence or positive regression dependence (PRDS); under arbitrary or negative dependence use (Benjamini & Yekutieli 2001 29:1165).
