| name | internal-structure-validation |
| description | Use when examining the internal structure of a selection procedure as supporting validity evidence — the relationships among items/components and how they conform to the intended conceptual framework (unidimensional vs. multidimensional). Covers dimensionality, factor-analytic models, when coefficient alpha is and isn't appropriate, and validity of subscores. Triggers: "internal structure", "dimensionality", "factor analysis / CFA", "coefficient alpha", "is this scale unidimensional", "subscale validity". |
| version | 1.0.0 |
| author | OpenMatter-Network |
| license | MIT |
| category | research |
| tags | ["Community","io-psychology","personnel-selection"] |
| permissions | [] |
Internal-structure validation
Evidence about how the components of a procedure (items, tasks, subscales) relate to one another
and conform to the conceptual framework that defines the procedure. Useful for planning and
developing a procedure and for supporting interpretation of total and subscores.
The critical limitation — state it up front
Internal-structure evidence does not, by itself, establish job relatedness. It shows the
procedure measures the intended construct(s); it does not show those scores relate to work
behavior or outcomes. Job relatedness still requires linking scores to the work via
criterion-related-validation, content-based-validation, or generalizing-validity-evidence.
Treat internal structure as support, not a stand-alone case.
Match the analysis to the conceptual framework
The relevant analyses depend on the procedure's intended structure:
- Single dimension/construct intended → offer evidence that covariances among components are
accounted for by a strong single factor. Coefficient alpha / internal consistency can be
appropriate evidence here.
- Multidimensional structure intended (hypothesized multifactor measure) → carefully examine
dimensionality (e.g., confirmatory factor analysis fitting the proposed structural model).
Overall internal consistency may be inappropriate as the index of choice.
Item inclusion should be driven primarily by relevance to the construct/content domain and only
secondarily by intercorrelations. Well-constructed components with near-zero correlations with
other components, scales, or the total score should not automatically be eliminated — if the
procedure deliberately spans different construct/content domains (e.g., a battery of a reading test,
an in-basket, and an interview), low inter-component correlations are expected and fine.
Cautions on coefficient alpha
High internal consistency can be misleading:
- A long, multi-dimensional measure can show high alpha simply because of the number of items
(Cortina, 1993), masking multidimensionality.
- A performance-rating form with theoretically unrelated scales can show high alpha because of
halo effect, not true unidimensionality.
So a high alpha is not proof of a single construct. Generic internal-consistency indices do not
evaluate internal structure for multidimensional procedures.
Subscores
When a multidimensional structure is proposed, evidence supporting inferences about subcomponent
scores may be needed if those subscores are interpreted or used. Don't report/interpret subscores you
haven't supported.
Process
- State the conceptual framework: how many dimensions/constructs, and how components map to
them.
- Choose analyses that fit that framework (single-factor evidence vs. CFA/dimensionality study).
- Select components by construct/content relevance first; don't purge low-correlating items
that belong conceptually.
- Use alpha only where a single dimension is intended; otherwise examine dimensionality
directly and beware item-count and halo artifacts.
- Support any subscores you intend to interpret.
- Remember to establish job relatedness separately.
Pitfalls
- Treating internal-structure evidence as establishing job relatedness — it does not; that
requires a separate source linking scores to work behavior/outcomes.
- Reporting a high coefficient alpha as proof of unidimensionality (item-count and halo artifacts).
- Purging well-constructed components that correlate near-zero with others when they belong
conceptually to a deliberately multidimensional procedure.
- Using overall internal consistency as the index of choice for a hypothesized multifactor measure.
- Interpreting or reporting subscores without supporting evidence for those subcomponent inferences.
Checklist
See also
criterion-related-validation · content-based-validation · generalizing-validity-evidence ·
fairness-and-bias-analysis (DIF is an item-level structural concern) · technical-validation-report
Source: Principles (5th ed., 2018), "Sources of Validity Evidence → Evidence of Validity Based on
Internal Structure."
