validation-planning

name	validation-planning
description	Use when planning or scoping a personnel-selection validation effort BEFORE data collection — to define the organization's needs, objectives, and constraints, specify the proposed uses and inferences, decide whether existing evidence suffices or new evidence is needed, and choose a validation strategy (criterion-related, content, internal structure, or generalization). Triggers: "plan a validation study", "which validation strategy", "do we need a local validity study", "scope a selection project", "justify a test for this job".
version	1.0.0
author	OpenMatter-Network
license	MIT
category	research
tags	["Community","io-psychology","personnel-selection"]
permissions	[]

Validation planning

The front end of any selection project. Validation should begin with a clear statement of the proposed uses and the intended interpretations and outcomes — then a design is chosen to support those inferences. Getting this wrong wastes the whole study: a poorly executed effort can lead an employer to reject a beneficial procedure or adopt an invalid one. "Misleading, poorly designed validation efforts should not be undertaken."

What validity means here

Validity is the degree to which evidence and theory support the interpretation of scores for a proposed use. You validate an inference (e.g., "scores predict job performance"), not a test. Plan around the inference you must defend.

The Standards name five sources of validity evidence; for employment the first three usually carry the argument:

1. Relationships with other variables (test–criterion) — see criterion-related-validation
2. Content — see content-based-validation
3. Internal structure — see internal-structure-validation
4. Response processes (often irrelevant to employment use)
5. Consequences (relevant to validity only when a negative consequence traces to a measurement property of the procedure — e.g., bias; mere subgroup-mean differences are not, by themselves, evidence against validity)

Sources are not competing "types" of validity; they are complementary evidence. Plan to combine them where the inference is complex, novel, or high-stakes.

Steps

1. Define the organization's needs, objectives, and constraints. Work collaboratively with HR, recruiting, legal/compliance, IT, labor relations, and affected stakeholders. Different units have competing objectives (rigor vs. applicant flow). Run a cost–benefit lens over candidate procedures.
2. Specify the proposed use(s) precisely. Target job(s) or job family, candidate pool (experienced vs. inexperienced; applicants vs. incumbents), the decision (hire, promote, place, certify), and how scores will be used (rank order, cutoff, band). The use drives the design.
3. Characterize the setting. One homogeneous organization or many? Stable work or rapidly changing? Workforce size and applicant availability — these constrain whether a local study is even feasible and which strategies (local criterion study, VG, synthetic, content) are open.
4. Inventory existing evidence. Is there sufficient accumulated validity evidence (meta-analytic bases for cognitive ability and a growing set of noncognitive measures; prior local studies; transportable studies) to support the proposed use without new data? Weigh the informational value of new evidence against its cost. Existing evidence alone may or may not suffice — judge the strength of the generalization to your setting. See generalizing-validity-evidence.
5. Apply the requirements of sound inference. Reliable and relevant measures, representative samples, appropriate analyses, controls over plausible confounds, and people qualified for the tasks they perform in the study.
6. Assess feasibility. Time, cost, sample size/statistical power, organizational disruption, union–management relations. Constraints may legitimately narrow the scope of defensible generalizations — but never justify a misleading design.
7. Choose a validation strategy that fits objectives, constraints, and the procedures/criteria. Often more than one source of evidence is valuable. Three illustrative fits:
   • Small population, strong cumulative evidence in similar settings → validity generalization.
   • Same job extended from one business unit to another → transportability.
   • Position unique to the organization, no external evidence → content-based strategy.
8. Communicate the plan. Management and workers need the purpose, the research plan, and their roles. Decide and communicate confidentiality (e.g., concurrent study data kept out of employment decisions). Document the plan.

Decision guidance: which strategy?

Situation	Lean toward
Adequate sample, relevant criterion obtainable, need local proof of prediction	criterion-related
KSAOs/behaviors sampled directly from work; content closely mirrors the job	content-based
Strong external cumulative evidence; small local sample	meta-analytic validity generalization
Same job already validated in a comparable unit	transportability
Many jobs share common work components	synthetic / job-component validity
Multidimensional procedure whose internal structure must be defended	internal structure (as support)

Pitfalls

• Designing the study before pinning down the inference and use.
• Treating subgroup-mean differences as automatic evidence against validity (they trigger scrutiny for bias, not a verdict).
• Underpowered local studies: adequate power while controlling Type I error often needs samples that are hard to obtain — consider this before committing to a criterion-related design.
• Letting convenience (available criterion, available sample, available software) drive design.
• Promising legal defensibility — the Principles address psychological method, not the law.

Checklist

• Proposed use(s), target job(s), and candidate pool stated explicitly
• Inference(s) to be supported written down
• Existing evidence inventoried; gap (if any) justified
• Feasibility (power, cost, time, access) assessed honestly
• Strategy chosen and rationale documented
• Stakeholders identified; communication & confidentiality plan set
• Plan documented (feeds technical-validation-report)

See also

work-analysis · criterion-related-validation · content-based-validation · internal-structure-validation · generalizing-validity-evidence · technical-validation-report

Source: Principles (5th ed., 2018), "Overview of the Validation Process" and "Operational Considerations → Initiating a Validation Effort / Selecting the Validation Strategy."