| name | content-based-validation |
| description | Use when building validity evidence by linking selection-procedure content to the work domain — demonstrating that the procedure samples important work behaviors, activities, and/or worker KSAOs defined by an analysis of work. Covers defining the content domain, SME qualifications and linkage judgments, sampling the domain, fidelity/specificity, competency-model foundations, and evaluating content evidence. Triggers: "content validity", "content-based strategy", "work sample", "job knowledge test", "link test content to the job", "SME linkage ratings", "no criterion data available". |
| version | 1.0.0 |
| author | OpenMatter-Network |
| license | MIT |
| category | research |
| tags | ["Community","io-psychology","personnel-selection"] |
| permissions | [] |
Content-based validation
Evidence that the content of a selection procedure is a representative sample of important work
behaviors, activities, and/or worker KSAOs defined by an analysis of work. The inference rests on
the fidelity of the procedure's content to the work — not on an empirical predictor–criterion
correlation. Often the strategy of choice when a job is unique, samples are too small for a
criterion study, or the procedure is built directly from the work (e.g., work samples, job-knowledge
tests).
When content evidence is and isn't the right tool
- Strong fit when procedure content is closely linked to job content.
- A construct labeled an ability or personality characteristic does not by itself rule out a
content strategy — but when content is less clearly linked to job content, other sources of
evidence take precedence (e.g., criterion-related, internal structure).
- Content evidence supports construct interpretation and generalization to job performance/
training when the procedure samples the work well.
SMEs define the domain — qualify them
The study's success hinges on SME qualifications. SMEs:
- define the work domain (important behaviors, activities, worker KSAOs),
- participate in the analysis of work, and
- judge the link between selection-procedure content and work/worker requirements.
SMEs must be competent to perform the work and knowledgeable about relevant organizational
characteristics (shift, location, equipment, software). For behavior definition, include people
fully knowledgeable about the work. Standardize and document the method for translating SME
judgments into the procedure; if SME ratings evaluate the content–work match, standardize the rating
procedures and criteria for each aspect.
Process
- Confirm feasibility. Are the work and worker requirements reasonably stable? Can
irrelevant content be minimized? Are qualified, unbiased SMEs available? Do cost/time allow the
needed fidelity?
- Define the content domain from accurate, thorough work information (behaviors, activities,
responsibilities, prerequisite KSAOs). The domain need not include everything done on the
job. State which important elements are included, how each links to the procedure, and why any
parts were excluded. Ground the definition in organizational needs, labor markets, and
specifications.
- Decide what the candidate must already have vs. will be trained. Base the procedure on the
analysis that defines the balance between KSAOs expected at entry and training the
organization provides. (Read-the-manual example: if employees are trained on the technical
manuals, test reading ability to the required level — not manual-specific knowledge.)
- Develop or choose the procedure; sample the domain. Random sampling of the domain is usually
neither feasible nor appropriate. Instead, sample the most important behaviors/KSAOs or those
prerequisite to others, or a smaller set predictive of critical outcomes. Document the
rationale in a test plan specifying which KSAOs are measured by which methods.
- Set the level of specificity deliberately. More fidelity to exact job components → stronger
content evidence, but less resilience to change and less transferability. If the work is
likely to change, build less job-specific content (e.g., software principles/techniques
rather than one program). Measure skills and knowledge that are job-relevant, never irrelevant
content, even when a full simulation isn't feasible (a single-step problem may substitute for a
multistep work sample).
- Establish administration and scoring guidelines in advance — especially when the procedure
includes work behaviors or activities that raise administration/scoring challenges.
- Evaluate the content evidence and document everything.
Evaluating content evidence
Content evidence rests on showing the procedure adequately samples and is linked to the
important work behaviors/activities/KSAOs from the work analysis. The documented development methods
are the primary evidence. The sufficiency of the match is a matter of professional judgment
based on evidence collected in the effort. Determine reliability of performance on the procedure
when feasible, choosing a reliability type that fits the measurement design and intended use (e.g.,
pass/fail vs. rank ordering).
Experience and training requirements as content
A procedure may include evidence of specific prior training, experience, or achievement — judged by
the relationship between the content of that experience and the content of the work. Superficial
resemblance is not enough: course titles and job titles may poorly indicate actual content or
proficiency. Evaluate the similarity of behaviors/activities/processes/KSAOs, not labels.
Competency modeling
A competency model can found a content study only if it is detailed and rigorous — as rigorous
as a traditional work-analysis project. Judge that rigor before relying on it. See work-analysis.
Pitfalls
- Including content not part of the work domain (standardization artifacts aside) or, worse,
irrelevant content.
- Over-specifying content for work that is about to change.
- Resting on a thin competency model.
- Treating face validity as content evidence (acceptability ≠ validity).
- Failing to document why domain elements were excluded.
Checklist
See also
work-analysis (defines the domain) · validation-planning ·
internal-structure-validation · generalizing-validity-evidence ·
technical-validation-report
Source: Principles (5th ed., 2018), "Sources of Validity Evidence → Evidence for Validity Based on
Content."
