| name | generalizing-validity-evidence |
| description | Use when justifying a selection procedure with validity evidence gathered elsewhere instead of (or alongside) a local study — via transportability, synthetic/job-component validity, or meta-analytic validity generalization (VG). Covers when each applies, the work-analysis link required, moderators, and limits on generalization. Triggers: "validity generalization", "use existing/meta-analytic evidence", "transport a study", "synthetic validity", "job component validity", "do we need a local study", "borrow validity". |
| version | 1.0.0 |
| author | OpenMatter-Network |
| license | MIT |
| category | research |
| tags | ["Community","io-psychology","personnel-selection"] |
| permissions | [] |
Generalizing validity evidence
When accumulated evidence is strong enough, you may justify a procedure in a new setting without a
local criterion study — by demonstrating generalized validity and making a compelling argument
for direct applicability to your situation. Three strategies (not mutually exclusive, not
exhaustive): transportability, synthetic/job-component validity, and meta-analytic
validity generalization. All require an analysis of the work to establish the link.
1. Transportability
Apply a specific procedure in a new situation based on a validation study conducted elsewhere,
when key similarities make that evidence applicable.
- Carefully review the original study for technical soundness and conceptual/empirical relevance
to the new situation.
- Compare on job content, job requirements, job context, and (if feasible) the applicant group.
- Document the comparison between the original validation sample/setting and your target use.
2. Synthetic validity / job-component validity
Justify use based on the validity of inferences for one or more components (job components) of
the work, established for those components, then "synthesized" (empirically combined) for a given job
or job family.
- Requires a detailed analysis of work that decomposes jobs into components and identifies which
components a job comprises.
- Powerful when many jobs share common components — provides a validity source where a separate
criterion study per job isn't feasible, and reduces burdensome data collection.
- Caveat: job requirements unique to a job may not be well covered by components and may need
other evidence.
3. Meta-analytic validity generalization (VG)
Cumulate validity findings across studies to estimate the predictor–criterion relationship for the
domains/settings of interest, and to determine how specific/generalizable the relationship is.
- VG shows that much of the observed variation in validity across settings is due to statistical
artifacts (sampling error, range restriction, criterion unreliability). Well established for
cognitive ability; accruing for several noncognitive measures.
- Organize around constructs. Generalization is straightforward when results are organized by
predictor and criterion constructs. Method labels alone (e.g., "interview," "SJT," "biodata")
are not constructs — a method can be designed to assess very different constructs, so you can
only generalize to applications of the method that share the relevant features (content, scoring,
meaning of scores).
- Evaluate the meta-analysis itself: methods and assumptions, their tenability, statistical
artifacts that could bias results, and moderators. When a substantive moderator is plausible,
consider statistical power and the precision of reported effects to detect it. Insist on full
reporting of how studies were categorized and analyzed; missing/unreported information undermines
the inference.
Local study vs. generalized evidence
Generalized evidence is often more useful than a single small local study. But a competent
local study with a large, organizationally relevant sample using the same test for the same work
can be more accurate and informative than an accumulation of small, heterogeneous, or deficient
studies that don't represent your setting. Watch for representation gaps: if your setting (e.g.,
a managerial job) isn't represented in the meta-analytic database (e.g., limited to entry-level
jobs), a local study may be more relevant. A Bayesian approach can formally combine meta-analytic
priors with locally estimated coefficients (Newman, Jacobs, & Bartram, 2007).
Sole reliance on cumulative evidence may also be insufficient for operational needs (e.g.,
optimal placement, combining procedures in a broader system) — supplementary local or cooperative
studies may still be warranted.
When generalization is NOT warranted
- Generalizing from a method studied for one construct to the same method used for a different
construct (e.g., technical-knowledge interviews → interpersonal-skill interviews).
- Applying meta-analytic results from one set of procedures/settings to a new setting using
different, unspecified procedures.
- Assuming predictor/criterion measures sharing a construct label are interchangeable without
rational and empirical support.
Pitfalls
- Skipping the analysis of work that links the borrowed evidence to your job(s) — required for all
three strategies.
- Organizing meta-analytic evidence by method label ("interview," "SJT") instead of by construct.
- Treating generalized evidence as automatically superior, ignoring representation gaps (your
setting absent from the meta-analytic database) where a local study would be more relevant.
- Transporting a study without reviewing its technical soundness and comparing job content,
requirements, context, and applicant group.
- Relying solely on cumulative evidence when operational needs (placement, combining procedures)
require local data.
Checklist
See also
validation-planning · work-analysis (required link) · criterion-related-validation
(local alternative/supplement) · fairness-and-bias-analysis · technical-validation-report
Source: Principles (5th ed., 2018), "Generalizing Validity Evidence."
