shiny-statistics-redesign

name	shiny-statistics-redesign
description	Statistics tab redesign patterns: 5-section narrative layout, BH correction for stratified tests, plotly legend positioning, synthetic union row density fix, test type UX guidance
author	smith6jt
date	"2026-03-06T00:00:00.000Z"

Shiny Statistics Tab Redesign - Research Notes

Experiment Overview

Item	Details
Date	2026-03-06
Goal	Redesign Statistics tab from flat 7-card grid to 5-section narrative layout with proper multiple testing correction, improved plotly legend placement, synthetic union row density fix, and test type UX guidance
Environment	R 4.x, Shiny 1.12.1, plotly 4.12.0, ggplot2, dplyr, scales
Status	Success

Context

The Statistics tab originally used a flat 7-card grid layout that was overwhelming and lacked narrative flow. Several statistical issues needed fixing: raw p-values from per-bin tests were not corrected for multiple comparisons across bins, plotly legends overlapped x-axis titles, synthetic union rows in prep_data() had NA density values causing data loss, and users had no guidance on when to use parametric vs non-parametric tests.

Verified Workflows

1. Five-Section Narrative Layout

Replaced the flat 7-card grid with numbered sections that guide the user through a statistical analysis workflow:

1. Configure -- Feature/region/test type selection with inline guidance
2. Primary Results -- Global test + pairwise table + forest plot
3. Size-Dependent Patterns -- Per-bin heatmap + trend analysis
4. Confounders -- Demographics (age, gender) stratification
5. Methods & Interpretation -- Dynamic text describing all tests, corrections, assumptions

Each section uses a section_heading() helper for consistent styling:

section_heading <- function(number, title, subtitle = NULL) {
  tagList(
    div(style = "display: flex; align-items: center; gap: 12px; margin-top: 20px; margin-bottom: 12px; border-bottom: 2px solid #e0e0e0; padding-bottom: 8px;",
      span(style = paste0(
        "background: linear-gradient(135deg, #4477AA, #5599CC); color: white; ",
        "font-weight: bold; border-radius: 50%; width: 30px; height: 30px; ",
        "display: flex; align-items: center; justify-content: center; font-size: 15px;"
      ), number),
      div(
        h4(style = "margin: 0; font-size: 17px; font-weight: 600;", title),
        if (!is.null(subtitle)) tags$small(style = "color: #666; font-size: 13px;", subtitle)
      )
    )
  )
}

Key design choices:

• Gradient pill badge (#4477AA to #5599CC) for section numbers
• 17px title, 13px subtitle
• Bottom border separates sections visually
• Numbered flow creates a narrative: configure -> test -> explore patterns -> check confounders -> understand methods

2. Plotly Legend Overlap Fix

When converting ggplot2 to plotly, legend positioning requires coordination between both systems:

# Step 1: Suppress ggplot legend (plotly will override it anyway)
g <- g + theme(legend.position = "none")

# Step 2: In plotly layout, create space and position legend below
p <- ggplotly(g) %>%
  layout(
    margin = list(b = 60),  # 60px bottom margin for legend room
    legend = list(
      orientation = "h",
      x = 0.5,
      xanchor = "center",
      y = -0.25
    )
  )

Critical: The margin(b = 60) is essential. Without it, plotly does not auto-expand to accommodate the legend, and it overlaps the x-axis title. The y = -0.25 positions the legend below the plot area, and xanchor = "center" centers it horizontally.

3. BH Correction Across Bins

Per-bin hypothesis tests (ANOVA, Kruskal-Wallis, Kendall) produce one p-value per bin. With 7 bins, uncorrected testing at alpha=0.05 inflates the family-wise error rate to ~30%. Apply Benjamini-Hochberg correction across all bins:

# In per_bin_anova() or per_bin_kendall():
results$p_raw <- results$p_value
results$p_adjusted <- p.adjust(results$p_value, method = "BH")

# Heatmap shows -log10(q) not -log10(p)
z_matrix <- -log10(results$p_adjusted)

# Star annotations use corrected values
stars <- ifelse(results$p_adjusted < 0.001, "***",
         ifelse(results$p_adjusted < 0.01, "**",
         ifelse(results$p_adjusted < 0.05, "*", "")))

The heatmap x-axis shows bin midpoints, y-axis shows test type, and z-values are -log10(q) where q is the BH-adjusted p-value. This ensures that significance claims account for the number of bins tested.

4. Synthetic Union Row Density Fix

prep_data() synthesizes union (Core+Peri) rows for the Microenvironment analysis mode. The original code set area_um2, region_um2, and area_density to NA for these rows:

# BROKEN: density is NA for synthetic union rows
union_row$area_um2 <- NA_real_
union_row$region_um2 <- NA_real_
union_row$area_density <- NA_real_

When the default metric is "Density" (which uses area_density), the !is.na(value) filter removes all synthetic rows. Only donors with pre-existing union rows in the source data survive -- typically 3 out of 15 donors.

Fix: Sum core + band areas and compute density:

# FIXED: compute actual values for synthetic union rows
union_row$area_um2 <- core_row$area_um2 + band_row$area_um2
union_row$region_um2 <- core_row$region_um2 + band_row$region_um2
union_row$area_density <- union_row$area_um2 / union_row$region_um2

5. Equal-Height Cards in a Row

When placing multiple cards side-by-side (e.g., pairwise table and forest plot), use flexbox to equalize heights:

fluidRow(
  style = "display: flex; flex-wrap: wrap;",
  column(5, div(style = "height: 100%;", pairwise_table_card)),
  column(7, div(style = "height: 100%;", forest_plot_card))
)

The display: flex; flex-wrap: wrap; on the row makes child columns stretch to equal height. Inner cards with height: 100% fill their column.

6. Test Type Explanation UX

Inline tags$small() below radio buttons explains parametric vs non-parametric assumptions so users can make informed choices:

radioButtons(ns("test_type"), "Test Type",
  choices = c("Non-parametric" = "nonparametric", "Parametric" = "parametric"),
  selected = "nonparametric", inline = TRUE
),
tags$small(style = "color: #666; font-size: 12px; display: block; margin-top: -5px;",
  "Non-parametric (Kruskal-Wallis + Wilcoxon): no normality assumption.",
  tags$br(),
  "Parametric (ANOVA + t-test): assumes normal distribution within groups."
)

Failed Attempts (Critical)

Attempt	Why it Failed	Lesson Learned
Plotly legend at y = -0.15 with no extra margin	Overlaps x-axis title -- plotly does not auto-expand to accommodate legend	Must set margin = list(b = 60) in layout() to create space, then position legend at y = -0.25
Running per-bin tests with raw p-values and star annotations at 0.05/0.01/0.001	Multiple testing across 7 bins inflates false positive rate to ~30%. A feature with no real effect will show spurious significance in 1-2 bins by chance	Must BH-correct p-values across all bins. Heatmap shows -log10(q) not -log10(p). Stars use corrected values.
Setting area_density = NA_real_ for synthetic union rows	Default metric is "Density" which uses area_density. NA values filtered by !is.na(value). Only donors with pre-existing union data survive (3/15).	Synthetic union rows must compute real values: sum core+band areas, divide for density. Never use NA for a metric that will be the default display.
Using legend.position = "bottom" in ggplot alone	ggplotly() overrides ggplot legend position. The ggplot "bottom" setting has no effect after conversion.	Set legend.position = "none" in ggplot theme, then control legend entirely via layout(legend = ...) in plotly.
Forest plot crammed into same card as pairwise table	Forest plot needs vertical space (350px minimum) to show confidence intervals clearly. Combining with table compresses both.	Give forest plot its own card. Use col-5 for table, col-7 for forest plot (wider for CI whiskers).
Flat 7-card layout for statistics	Users couldn't find a logical reading order. Cards competed for attention. No clear workflow from "configure" to "interpret".	Numbered sections with visual hierarchy (gradient badge, subtitle, border) create a narrative flow. Users progress from configuration through results to methods naturally.

Key Parameters

Parameter	Value	Context
Plotly legend bottom margin	b = 60 pixels	layout(margin = list(b = 60))
Plotly legend y position	-0.25	layout(legend = list(y = -0.25))
Section heading gradient	#4477AA to #5599CC	linear-gradient(135deg, ...)
Section title font size	17px	h4 with font-size: 17px
Section subtitle font size	13px	tags$small with font-size: 13px
Forest plot height	350px	Minimum for readable CI whiskers
Table:forest column split	5:7	column(5, table) + column(7, forest_plot)
BH correction method	p.adjust(method = "BH")	Applied across all bins per test type
Per-bin minimum observations	3	Bins with <3 points skipped in per-bin tests

Key Insights

• Narrative > Grid: A numbered sequence (Configure -> Test -> Explore -> Check -> Understand) is more usable than a flat grid of cards. Users have a clear path through the analysis.
• Multiple testing is mandatory for binned analysis: 7 bins at alpha=0.05 gives ~30% family-wise error rate. BH correction is the minimum -- Bonferroni is too conservative for correlated bins.
• Default metric must never be NA: If the default display metric is computed from other fields, synthetic/derived rows must compute it too. NA defaults silently filter out data with no user-visible error.
• Plotly legend needs explicit space: Unlike ggplot2, plotly does not auto-allocate margin for legends placed outside the plot area. Always pair margin(b=N) with legend(y=-M).
• Separate ggplot and plotly legend control: When using ggplotly(), set legend.position = "none" in ggplot to suppress it, then control legend entirely through plotly's layout(). Trying to use ggplot's legend positioning with plotly conversion produces unpredictable results.
• Inline test guidance reduces support burden: A two-line explanation of parametric vs non-parametric assumptions below the radio buttons prevents the most common user confusion without requiring external documentation.
• Forest plots need dedicated space: Confidence interval whiskers are the primary visual element. Cramming the forest plot into a shared card with the pairwise table compresses both below usefulness. Minimum 350px height in its own card.

References

• Benjamini & Hochberg (1995): Controlling the false discovery rate
• Shiny p.adjust(): https://stat.ethz.ch/R-manual/R-devel/library/stats/html/p.adjust.html
• Plotly R layout: https://plotly.com/r/reference/layout/
• Related skill: shiny-feature-patterns (pseudo-log, AUC, effect sizes, shared sidebar)
• Related skill: shiny-modularization (module extraction, namespacing)
• Islet Explorer: app/shiny_app/R/mod_statistics_server.R, app/shiny_app/R/mod_statistics_ui.R, app/shiny_app/R/utils_stats.R

Demographics Card Redesign (2026-03-16)

Verified Patterns:

• Islet-level age scatter (alpha=0.4, size=1.5) with Pearson r subtitle noting "correlated within donors" — more informative than 15 donor-mean points
• Sex box plot (facet_wrap(~ gender)) replaces p-value table — shows distribution shape, not just significance
• AAb analysis (Aab+ only): n_aab = rowSums(sapply(aab_avail, ...)) counts positive autoantibodies per islet; box plot by AAb count with Kruskal-Wallis test
• Full-width demographics card with side-by-side age + sex plots via fluidRow(column(6), column(6)) inside renderUI
• AUC card belongs in Primary Results (Section 2), not Confounders — it's a primary outcome metric

Layout Lessons:

• 3-column equal-height: fluidRow(style = "display: flex; flex-wrap: wrap;") with column(4) + height: 100% cards
• Trend plot must match heatmap card height — use height: 100% on both card divs + same plotlyOutput height
• Trend plot legend: remove ggplot legend.position = "none" suppression, add plotly showlegend = TRUE with legend(orientation = "h", y = -0.15)

Failed Approaches:

What failed	Why	Fix
Global Kendall τ arrow in per-bin trend title	Misleading — single global τ doesn't represent per-bin variation	Remove title entirely; explain disease coding (ND=0, Aab+=1, T1D=2) in card description
"Wider bins recommended" in UI text	Implementation jargon, not interpretive guidance	Replace with explanation of what τ values mean
Gender terminology	"Gender" is a social construct; donor metadata records biological sex	Use "Sex" in all user-facing labels; keep gender as internal variable name

Mixed-Effects Model Fix (2026-03-17)

Problem: lmer_test_donor() used car::Anova(fit, type="III") to extract the mixed-effects p-value, looking for "Pr(>Chisq)" in the result table. This silently returned NULL → p-value always N/A. The donor-level ANOVA worked fine, making it appear the mixed-effects test was broken.

Root Cause: car::Anova with Type III on lmerMod objects uses Wald chi-square tests and the column name Pr(>Chisq). With only one fixed effect (donor_status), Type I = Type III, making car unnecessary.

Fix: Replace car::Anova(fit, type="III") with lmerTest::anova(fit) which uses Satterthwaite F-tests and the standard Pr(>F) column. Also wrap the model fit itself in tryCatch and return an error_msg field so failures are visible in the UI (yellow warning banner) instead of silently producing N/A.

Additional Fix: lmerTest 3.2-1 was listed as a dependency but not actually installed. Always verify package availability, not just configuration.

Verified Patterns:

• Trend plot legend: margin(b=100) + legend(y=-0.3) prevents overlap with x-axis title (previous b=70, y=-0.15 was too tight)
• Demographics card equal spacing: wrap sub-sections in div(style="margin-bottom:18px") instead of using hr dividers
• AAb table + plot side-by-side: fluidRow(column(6, table), column(6, plot)) instead of stacking vertically
• Hypothesis table labels: avoid R package names ("lmer", "emmeans") in user-facing text. Use descriptive labels: "mixed-effects, donor random intercept", "mixed-effects pairwise", "donor-level means"

Failed Approaches (2026-03-17):

What failed	Why	Fix
car::Anova(fit, type="III") for mixed-effects p-value	Column name mismatch (Pr(>Chisq) vs expected), and Type III unnecessary with one fixed effect	Use lmerTest::anova(fit) which returns Pr(>F) via Satterthwaite F-test
Silent N/A for model failures	tryCatch(lmer_test_donor(rdf), error=function(e) NULL) swallows all errors, user sees only "N/A"	Return error_msg field from function, display yellow warning banner in UI
R package jargon in table labels ("lmer", "emmeans", "Sensitivity")	Users don't know what "lmer" means or why something is called "sensitivity"	Use plain descriptions: "Overall (mixed-effects, donor random intercept)", "Donor-level means (ANOVA, N=15 donors)"
Trend plot legend at y=-0.15, margin(b=70)	Still overlaps x-axis title text on narrow viewports	Increase to y=-0.3, margin(b=100)