scientific-critical-thinking

name	scientific-critical-thinking
description	Evaluating scientific evidence and claims. Covers study design hierarchy (RCT to expert opinion), effect sizes (OR, RR, NNT, Cohen's d), confounding, p-value vs clinical significance, GRADE quality assessment, reproducibility, and bias types (selection, information, confounding, reporting). Use when reading a paper or assessing claims.
license	CC-BY-4.0

Scientific Critical Thinking: Evaluating Evidence and Claims

Overview

Scientific critical thinking is the disciplined application of logical and methodological standards to evaluate whether a study's design, analysis, and interpretation support its conclusions. It is the skill that separates a researcher who synthesizes evidence from one who accumulates it. This guide covers the hierarchy of evidence, the mechanics of common biases, effect size interpretation, the p-value controversy, GRADE evidence grading, and common logical fallacies in the interpretation of scientific literature.

Key Concepts

1. Study Design Hierarchy

Study designs vary in their ability to support causal inference. The hierarchy below applies to questions about the effect of an intervention or exposure on an outcome:

Systematic reviews and meta-analyses of RCTs (highest causal certainty)
    ↓
Randomized Controlled Trials (RCTs)
    ↓
Non-randomized controlled trials / cluster-randomized trials
    ↓
Prospective cohort studies (follow exposure → outcome forward in time)
    ↓
Retrospective cohort studies
    ↓
Case-control studies (compare exposed vs. unexposed given outcome)
    ↓
Cross-sectional studies (measure exposure and outcome simultaneously)
    ↓
Case series and case reports
    ↓
Expert opinion, mechanistic reasoning, animal models (lowest causal certainty)

Important exceptions: For questions about rare outcomes, case-control designs are often more efficient than cohort studies. For questions about diagnostic accuracy, randomized designs are usually inappropriate — cross-sectional or cohort designs with verified reference standards are preferred. For harm questions, RCTs are often infeasible (ethical constraints), making large cohort studies the best available evidence.