원클릭으로 Manus에서 모든 스킬 실행

alterlab-ssci-measurement-gate

스타46

포크6

업데이트2026년 7월 2일 13:45

Gates measurement quality before a scale or instrument is trusted — checks that each construct is defined and operationalized, that reliability is evidenced with McDonald omega (not Cronbach alpha alone, which assumes tau-equivalence and is only a lower bound), that reliability is not confused with validity (content, criterion, convergent/discriminant construct validity), and that measurement invariance is tested before comparing groups. Use when asking whether a scale or survey instrument is valid, reporting a Cronbach alpha, building or adopting a multi-item measure, or comparing a latent construct across groups. For designing the questionnaire items prefer alterlab-survey-design; for running the confirmatory factor analysis prefer alterlab-sem-psychometrics; to execute basic statistics prefer alterlab-statistical-analysis. Part of the AlterLab Academic Skills suite.

설치

Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.

Manus에서 실행

출처

AlterLab-IEU

AlterLab-IEU/AlterLab-Academic-Skills

GitHub 저장소 열기 Creator 저장소 보기

다운로드

Manus에서 실행

Measurement Gate — Reliability Is Not Validity

Skill type: DISCIPLINE-ENFORCING. Before any hypothesis test consumes a scale, this gate checks that the scale measures what it claims, reliably, and comparably across groups. It does not fit factor models itself — it requires the right evidence and routes execution to the psychometrics skill.

The Core Rule

A RELIABLE MEASURE OF THE WRONG CONSTRUCT IS STILL WRONG. RELIABILITY ≠ VALIDITY.

High internal consistency only means the items move together; it says nothing about whether they capture the intended construct. Evidence for validity (content, criterion, construct) is separate and mandatory. And the usual reliability number — Cronbach alpha — rests on an assumption (tau-equivalence: equal item loadings) that real scales rarely meet, so alpha is a lower bound that can under- or (with correlated errors) over-state reliability.

When to Use This Skill

"Is my scale / survey instrument valid?"
"My Cronbach alpha is 0.78 — is the scale good to use?" (← alpha alone is not enough)
"I built a new 6-item measure of burnout; how do I show it works?"
"I want to compare a latent trait (e.g. trust) across countries/groups."

Does NOT Trigger

The request is really about…	Route to	Why not this skill
Writing / wording the questionnaire items, response scales	`alterlab-survey-design`	Instrument construction, upstream of validation.
Running the CFA / SEM / IRT / invariance model	`alterlab-sem-psychometrics`	This gate requires CFA; that skill fits it.
Basic descriptive/inferential stats execution	`alterlab-statistical-analysis`	Computation, not measurement discipline.
Which statistical test to run on a measured outcome	`alterlab-test-selection-guard`	Test choice, not measurement quality.
Sampling frame / how many respondents	`alterlab-ssci-sampling-gate`	Sampling, not measurement.

Reliability: report alpha AND omega

Cronbach's alpha assumes tau-equivalence (all items load equally on one factor) and is a lower bound on reliability under that model. McDonald's omega relaxes tau-equivalence by weighting items by their factor loadings and should be reported alongside alpha (McNeish, 2018, Psychological Methods, "Thanks coefficient alpha, we'll take it from here"). In practice the alpha-vs-omega gap for well-constructed unidimensional scales is often small (Warne, 2025 — around 4.5% underestimate), so the discipline is: report both, plus item-total correlations and a dimensionality check — not to fetishize a single number, but to show the estimate is not resting on an unexamined assumption.

Rules this gate enforces:

Alpha alone is insufficient. Ask for omega; if only alpha is available, state the tau-equivalence caveat explicitly.
Reliability presupposes dimensionality. A high alpha on a multidimensional set is meaningless — check the factor structure (CFA, alterlab-sem-psychometrics) first.
Item diagnostics. Item-total correlations and (for two halves) Spearman-Brown; drop or revise items that do not cohere — but pre-specify such decisions, do not fish.

Validity: four kinds, none optional

Validity	Question	Evidence
Content	Do items cover the construct's domain?	expert review, blueprint against the definition
Criterion	Does it predict/correlate with a gold standard?	concurrent & predictive correlations
Convergent	Does it correlate with related measures?	AVE, correlations with kindred scales
Discriminant	Is it distinct from other constructs?	heterotrait-monotrait ratio, cross-loadings

Construct validity is established with a confirmatory factor analysis — route the fit to alterlab-sem-psychometrics (CFI/TLI, RMSEA, SRMR against Hu & Bentler cutoffs).

Measurement invariance before group comparison

Comparing a latent construct across groups (countries, waves, conditions) is only valid if the measure means the same thing in each group. Require a multi-group CFA invariance sequence — configural → metric → scalar — before interpreting group differences in means. Without at least metric/scalar invariance, a "difference" may be a measurement artifact, not a real effect. Route the test to alterlab-sem-psychometrics.

The Design Passport (hand-off)

Append to each construct in the Passport: reliability (alpha + omega + caveat), validity_evidence (which of the four kinds are shown), dimensionality (CFA fit), and invariance (level established) before any group comparison downstream.

Self-Check Before Advancing

Is each construct explicitly defined and operationalized to its items?
Is reliability shown with omega (or alpha with the tau-equivalence caveat stated)?
Is at least one form of validity evidence present — not just reliability?
Was dimensionality confirmed (CFA) before trusting a single reliability coefficient?
Is measurement invariance established before comparing the construct across groups?

References

references/reliability_and_validity.md — alpha vs omega math, the four validities, the invariance sequence, and reporting templates. Loaded on demand.

Part of the AlterLab Academic Skills suite.

이 저장소의 다른 Skills

같은 저장소

alterlab-skill-finder

AlterLab-IEU/AlterLab-Academic-Skills

The AlterLab front door and multi-agent launcher — routes a task to the right AlterLab skill(s) when the user invokes the suite without naming one, and for a multi-stage goal (or on the keyword 'alterflow', aliases 'alterresearch' / 'ultralab') it CLARIFIES the goal with a few questions, SELECTS the skills the task needs, and runs a dynamic multi-agent workflow composing them (via alterlab-workflow-orchestration, alterlab-research-pipeline, or alterlab-ssci-orchestrator). Triggers on 'use AlterLab skills', 'which AlterLab skill for X', 'is there an AlterLab skill for…', a multi-stage research goal, 'alterflow …', or any generic AlterLab request where the user does not know skill names. It always asks clarifying questions before executing a multi-step run. Use when someone references AlterLab generically, describes a multi-stage goal, or fires the alterflow keyword; when the user already names a specific skill, defer to that skill directly. Part of the AlterLab Academic Skills suite.

2026-07-0246

alterlab-meta-analysis

AlterLab-IEU/AlterLab-Academic-Skills

Runs quantitative meta-analysis — computes effect sizes (Hedges' g / standardized mean difference, log odds/risk ratios) with their variances, pools them under fixed-effect and random-effects models, quantifies heterogeneity (I-squared, tau-squared, Cochran's Q), draws forest and funnel plots, and tests publication bias (Egger's regression, trim-and-fill) — using statsmodels.stats.meta_analysis in Python or the field-standard R metafor via Rscript. It enforces PRISMA reporting and the random- vs fixed-effect decision. Use when pooling effect sizes across studies, running a systematic review's quantitative synthesis, or assessing heterogeneity and publication bias. For finding and screening the literature prefer alterlab-deep-research; for a single study's statistics prefer alterlab-statistical-analysis. Part of the AlterLab Academic Skills suite.

2026-07-0246

alterlab-missing-data

AlterLab-IEU/AlterLab-Academic-Skills

Handles missing data with principled methods — forces an explicit MCAR / MAR / MNAR mechanism statement, then applies multiple imputation by chained equations (MICE) with Rubin's-rules pooling of estimates and standard errors, or full-information maximum likelihood (FIML) where a likelihood/SEM model applies. Uses statsmodels MICE / MICEData in Python or the field-standard R mice via Rscript, and warns that single (mean/regression) imputation and scikit-learn's IterativeImputer return one completed dataset without Rubin's-rules pooling, so they understate standard errors if used as multiple imputation. Use when a dataset has missing values, when choosing an imputation strategy, or when reporting how missingness was handled. For general modeling on complete data prefer alterlab-statistical-analysis; for latent-variable models with FIML prefer alterlab-sem-psychometrics. Part of the AlterLab Academic Skills suite.

2026-07-0246

alterlab-multilevel-models

AlterLab-IEU/AlterLab-Academic-Skills

Fits and reports mixed-effects / multilevel / hierarchical models for clustered, nested, longitudinal, and repeated-measures data — random intercepts and slopes, variance components and the ICC, cross-level interactions, and GLMMs (logistic/Poisson) — using statsmodels MixedLM and bambi (Bayesian on PyMC) in Python, or the field-standard R lme4 / glmmTMB / brms via Rscript. It enforces the reporting items reviews find under-reported: full fixed + random specification, centering, variance components + ICC, estimation method, assumption checks, model comparisons, and effect sizes. Use when data are grouped/nested (students in schools, repeated measures, panel/longitudinal) and the question concerns within- vs between-cluster variation. For general single-level regression prefer alterlab-statsmodels; for panel fixed effects used for causal identification prefer alterlab-causal-inference. Part of the AlterLab Academic Skills suite.

2026-07-0246

alterlab-qualitative-analysis

AlterLab-IEU/AlterLab-Academic-Skills

Analyzes qualitative data as a dispatched pipeline module — codebook development, thematic / framework / content analysis, and inter-coder reliability computed correctly (Krippendorff's alpha as primary via the krippendorff package or a bundled stdlib nominal calculator with bootstrap CIs; Cohen's / Fleiss' kappa via statsmodels) with 95% CIs and thresholds (alpha >= .80 reliable, .667-.80 tentative). It BRANCHES by design: coefficient-based ICR for codebook / content-analytic coding, versus consensus-and-reflexivity for reflexive thematic analysis where a statistic is not the right criterion. Supports human-vs-LLM double-coding with an alpha check against a human gold standard. Use when coding interviews or open-ended text, building a codebook, or reporting intercoder reliability. For topic modeling / embeddings / supervised text classification prefer alterlab-text-as-data; for the reflexivity gate prefer alterlab-ssci-reflexivity-gate. Part of the AlterLab Academic Skills suite.

2026-07-0246

alterlab-ssci-orchestrator

AlterLab-IEU/AlterLab-Academic-Skills

Coordinates a social-science study through the stage-gated methods pipeline — question then design-gate then measurement-gate then sampling-gate then the right analysis module then inference-gate then reporting — holding a single YAML Design Passport that each gate reads and appends, and enforcing gate order with PASS / WARN / BLOCK semantics (design-gate and inference-gate are fail-closed). It is thin: it routes and holds the artifact, it does not run analysis itself. Use when a researcher describes a whole social-science study end to end, asks to run the methods pipeline or a full methodology review, or wants the design-to-inference workflow coordinated rather than a single step. For a single stage trigger that gate directly (alterlab-ssci-design-gate, -measurement-gate, -sampling-gate, -inference-gate); for multi-agent execution mechanics see alterlab-workflow-orchestration. Part of the AlterLab Academic Skills suite.

2026-07-0246

name	alterlab-ssci-measurement-gate
description	Gates measurement quality before a scale or instrument is trusted — checks that each construct is defined and operationalized, that reliability is evidenced with McDonald omega (not Cronbach alpha alone, which assumes tau-equivalence and is only a lower bound), that reliability is not confused with validity (content, criterion, convergent/discriminant construct validity), and that measurement invariance is tested before comparing groups. Use when asking whether a scale or survey instrument is valid, reporting a Cronbach alpha, building or adopting a multi-item measure, or comparing a latent construct across groups. For designing the questionnaire items prefer alterlab-survey-design; for running the confirmatory factor analysis prefer alterlab-sem-psychometrics; to execute basic statistics prefer alterlab-statistical-analysis. Part of the AlterLab Academic Skills suite.
license	MIT
allowed-tools	Read Bash(python:*)
compatibility	No API key required. A discipline-enforcing measurement-quality skill; reliability/CFA execution is handed to sibling skills that run locally via `uv run python`.
metadata	{"skill-author":"AlterLab","version":"1.0.0","depends_on":"alterlab-survey-design (item design), alterlab-sem-psychometrics (CFA/invariance), alterlab-statistical-analysis"}

Measurement Gate — Reliability Is Not Validity

The Core Rule

A RELIABLE MEASURE OF THE WRONG CONSTRUCT IS STILL WRONG. RELIABILITY ≠ VALIDITY.

When to Use This Skill

"Is my scale / survey instrument valid?"
"My Cronbach alpha is 0.78 — is the scale good to use?" (← alpha alone is not enough)
"I built a new 6-item measure of burnout; how do I show it works?"
"I want to compare a latent trait (e.g. trust) across countries/groups."

Does NOT Trigger

The request is really about…	Route to	Why not this skill
Writing / wording the questionnaire items, response scales	`alterlab-survey-design`	Instrument construction, upstream of validation.
Running the CFA / SEM / IRT / invariance model	`alterlab-sem-psychometrics`	This gate requires CFA; that skill fits it.
Basic descriptive/inferential stats execution	`alterlab-statistical-analysis`	Computation, not measurement discipline.
Which statistical test to run on a measured outcome	`alterlab-test-selection-guard`	Test choice, not measurement quality.
Sampling frame / how many respondents	`alterlab-ssci-sampling-gate`	Sampling, not measurement.

Reliability: report alpha AND omega

Rules this gate enforces:

Alpha alone is insufficient. Ask for omega; if only alpha is available, state the tau-equivalence caveat explicitly.
Reliability presupposes dimensionality. A high alpha on a multidimensional set is meaningless — check the factor structure (CFA, alterlab-sem-psychometrics) first.
Item diagnostics. Item-total correlations and (for two halves) Spearman-Brown; drop or revise items that do not cohere — but pre-specify such decisions, do not fish.

Validity: four kinds, none optional

Validity	Question	Evidence
Content	Do items cover the construct's domain?	expert review, blueprint against the definition
Criterion	Does it predict/correlate with a gold standard?	concurrent & predictive correlations
Convergent	Does it correlate with related measures?	AVE, correlations with kindred scales
Discriminant	Is it distinct from other constructs?	heterotrait-monotrait ratio, cross-loadings

Construct validity is established with a confirmatory factor analysis — route the fit to alterlab-sem-psychometrics (CFI/TLI, RMSEA, SRMR against Hu & Bentler cutoffs).

Measurement invariance before group comparison

The Design Passport (hand-off)

Self-Check Before Advancing

Is each construct explicitly defined and operationalized to its items?
Is reliability shown with omega (or alpha with the tau-equivalence caveat stated)?
Is at least one form of validity evidence present — not just reliability?
Was dimensionality confirmed (CFA) before trusting a single reliability coefficient?
Is measurement invariance established before comparing the construct across groups?

References

references/reliability_and_validity.md — alpha vs omega math, the four validities, the invariance sequence, and reporting templates. Loaded on demand.

Part of the AlterLab Academic Skills suite.