pairwise-ma-methodology

# Pairwise Meta-Analysis Methodology Comprehensive methodological guidance for conducting rigorous pairwise meta-analysis following Cochrane and PRISMA guidelines. ## When to Use This Skill - Planning a pairwise meta-analysis - Choosing between fixed and random effects models - Interpreting heterogeneity statistics - Assessing publication bias - Designing sensitivity analyses - Reviewing pairwise MA code or results ## Fixed vs Random Effects ### Decision Framework ``` Are studies functionally identical? ├── Yes → Fixed-effect model appropriate │ - Same population, intervention, comparator, outcome │ - Estimating single "true" effect │ └── No (usually the case) → Random-effects model - Studies differ in ways that affect true effect - Estimating mean of distribution of effects - More generalizable inference ``` ### When to Use Fixed-Effect - Studies are very similar (rare in practice) - Want to estimate effect in "identical" studies - Very few studies (< 5) - random effects unreliable - Sensitivity analysis alongside random effects ### When to Use Random-Effects - Studies differ in populations, settings, methods - Want inference applicable beyond included studies - Default choice for most meta-analyses - Use with appropriate adjustments (Knapp-Hartung) ### Key Differences | Aspect | Fixed-Effect | Random-Effects | |--------|-------------|----------------| | Assumption | Common true effect | Distribution of true effects | | Weights | Based on precision only | Includes between-study variance | | Small study | More weight | Less weight | | Large study | Less relative weight | More weight | | CI width | Narrower (if heterogeneity exists) | Wider (appropriately) | | Inference | To identical studies | To broader population | ## Heterogeneity Assessment ### Statistics Overview #### Q Statistic (Cochran's Q) - Tests null hypothesis of homogeneity - Follows chi-square distribution under null - Low power with few studies - Overpowered with many studies ```r # Interpretation Q_pvalue < 0.10 # Suggests heterogeneity (use 0.10, not 0.05) ``` #### I² (Inconsistency Index) - Percentage of variability due to heterogeneity (vs sampling error) - Independent of number of studies - Has wide confidence interval with few studies | I² Value | Interpretation | |----------|---------------| | 0-25% | Low heterogeneity | | 25-50% | Moderate heterogeneity | | 50-75% | Substantial heterogeneity | | >75% | Considerable heterogeneity | **Caution**: These thresholds are rules of thumb, not strict cutoffs. #### τ² (Tau-squared) - Absolute between-study variance - On scale of effect measure - Used for prediction intervals - Compare to typical effect sizes for context #### H² - Relative excess heterogeneity - H² = Q/(k-1) where k = number of studies - H² = 1 means no heterogeneity ### Prediction Intervals **Critical**: Always report prediction intervals alongside confidence intervals. - CI: Uncertainty in mean effect estimate - PI: Range where 95% of true study effects would lie ```r # In meta package metabin(..., prediction = TRUE) ``` If PI includes null but CI doesn't: - Mean effect is statistically significant - But future studies might show no effect or opposite effect - Heterogeneity is clinically important ## Investigation of Heterogeneity ### Subgroup Analysis ```r # Categorical moderator update(ma_result, subgroup = risk_of_bias) # Requirements: # - Pre-specified in protocol # - Limited number of subgroups # - Biological/clinical rationale # - Report within and between subgroup heterogeneity ``` ### Meta-Regression ```r # Continuous moderator rma(yi, vi, mods = ~ year + sample_size, data = es_data) # Requirements: # - Minimum 10 studies per moderator # - Pre-specified moderators # - Avoid overfitting # - Use Knapp-Hartung adjustment # - Permutation test for multiple moderators ``` ### Rule of Thumb for Investigation - Need ≥10 studies for meaningful subgroup analysis - Meta-regression requires even more studies - Pre-specify investigations in protocol - Report all investigated moderators (avoid selective reporting) ## Publication Bias Assessment ### Visual Assessment: Funnel Plot ```r funnel(ma_result) # Look for: # - Asymmetry (small studies with large effects) # - Missing studies in certain regions # - Outliers ``` ### Statistical Tests #### Egger's Test (Continuous Outcomes) ```r metabias(ma_result, method.bias = "linreg") # P < 0.10 suggests asymmetry # Low power with < 10 studies ``` #### Peters' Test (Binary Outcomes) ```r metabias(ma_result, method.bias = "peters") # Better for OR than Egger's ``` #### Begg's Rank Test ```r metabias(ma_result, method.bias = "rank") # Non-parametric alternative # Lower power than regression tests ``` ### Adjustment Methods #### Trim-and-Fill ```r trimfill(ma_result) # Imputes "missing" studies # Provides adjusted estimate # Sensitivity analysis, not definitive correction ``` #### Selection Models ```r # More sophisticated approaches # Model the selection process # Available in metafor and weightr packages ``` ### Interpretation Cautions - Asymmetry ≠ publication bias (could be true heterogeneity) - Tests have low power with few studies - Don't over-interpret with < 10 studies - Multiple causes of asymmetry exist ## Sensitivity Analyses ### Essential Sensitivity Analyses 1. **Fixed vs Random Effects** - Report both; if results differ, investigate why 2. **Leave-One-Out** ```r metainf(ma_result) # Identifies influential studies ``` 3. **Risk of Bias** - Exclude high risk of bias studies - Subgroup by risk of bias 4. **Influence Diagnostics** ```r influence(ma_result) # DFBETAS, Cook's distance ``` 5. **Different Effect Measures** - OR vs RR vs RD for binary - May give different conclusions 6. **Estimation Method** - DerSimonian-Laird vs REML vs ML ### GOSH Analysis ```r # Graphical display of study heterogeneity gosh(ma_result) # Identifies subsets with different results ``` ## Reporting Checklist (PRISMA) ### Methods - [ ] Effect measure and rationale - [ ] Model choice (fixed/random) and rationale - [ ] Heterogeneity measures planned - [ ] Publication bias assessment planned - [ ] Sensitivity analyses planned - [ ] Software and packages used ### Results - [ ] Number of studies and participants - [ ] Pooled effect with CI - [ ] Prediction interval - [ ] Heterogeneity statistics (Q, I², τ²) - [ ] Forest plot - [ ] Funnel plot (if ≥10 studies) - [ ] Publication bias test results - [ ] Sensitivity analysis results ## Common Pitfalls ### 1. Using Q p-value to Choose Model - Wrong: "Q p > 0.05, so use fixed-effect" - Right: Choose based on study similarity, report both ### 2. Ignoring Prediction Intervals - CI shows precision of mean estimate - PI shows variability in true effects - Both are clinically important ### 3. Over-interpreting I² - I² has wide CI with few studies - Context matters (clinical significance) - Don't use arbitrary thresholds mechanically ### 4. Selective Subgroup Analysis - Pre-specify in protocol - Report all, not just significant ones - Adjust for multiple testing ### 5. Publication Bias Assessment with Few Studies - Tests unreliable with < 10 studies - State this limitation, don't perform test ## Quick Reference Code ```r library(meta) # Basic random-effects MA (binary) ma <- metabin( event.e, n.e, event.c, n.c, studlab = study, data = dat, sm = "OR", method = "MH", method.tau = "REML", hakn = TRUE, # Knapp-Hartung adjustment prediction = TRUE # Prediction interval ) # Forest plot forest(ma, sortvar = TE, prediction = TRUE) # Funnel plot and Egger's test funnel(ma) metabias(ma, method.bias = "linreg") # Influence analysis metainf(ma) # Subgroup analysis update(ma, subgroup = risk_of_bias) ``` ## Resources - Cochrane Handbook: https://training.cochrane.org/handbook - PRISMA Statement: http://www.prisma-statement.org/ - Higgins & Green: Cochrane Handbook for Systematic Reviews - Borenstein et al.: Introduction to Meta-Analysis