| name | selection-decisions-and-scoring |
| description | Use when deciding how to combine selection procedures and turn scores into decisions — compensatory vs. multiple-hurdle models, cutoff scores, banding, rank-order/top-down selection, norms, and communicating effectiveness via expectancy charts and utility. Covers the validity/diversity tradeoffs and the documentation each choice requires. Triggers: "cutoff score", "banding", "rank order vs cutoff", "compensatory vs multiple hurdle", "combine test scores", "set a passing score", "utility analysis", "expectancy chart", "norms". |
| version | 1.0.0 |
| author | OpenMatter-Network |
| license | MIT |
| category | research |
| tags | ["Community","io-psychology","personnel-selection"] |
| permissions | [] |
Selection decisions and scoring
How scores become hiring/promotion decisions. Every choice here affects validity, expected
performance of those selected, and subgroup passing rates — so each requires a documented
rationale. There are few absolutes; professional judgment driven by organizational goals governs.
Combining procedures into a system
When multiple procedures form the basis of a decision, both the individual components and the
combination must be supported by validity evidence. Document the method and rationale for
combining and sequencing. Organizations weight differently depending on whether they emphasize
maximizing validity, minimizing subgroup differences, or balancing the two — state which.
Recall from criterion-related-validation: effective weights ≠ nominal weights (they depend on
component variances/covariances, and differential range restriction can distort them).
Compensatory vs. multiple hurdle
- Compensatory — candidates achieve a specified total across assessments; a high score on
one can offset a low score on another.
- Multiple hurdle — candidates must clear a score on each assessment (often sequential).
- Hybrids are possible (a hurdle on one critical predictor, compensatory among the rest).
No model is universally correct. The method of combining scores can affect the overall
reliability of the process and subgroup passing rates (Sackett & Roth, 1996). Present the
rationale and supporting evidence for the model recommended.
Cutoff scores vs. rank order
Two common strategies: a cutoff score (reject below a point) or rank-order / top-down
selection.
- There is no single best method for setting cutoffs. Options include criterion-referenced
cutoffs (when the predictor links to a meaningful performance threshold) and others (Mueller,
Norris, & Oppler, 2007).
- With valid predictors showing linearity, cutoffs may be set as high or low as needed to
meet organizational requirements. Nonmonotonicity in the predictor–criterion relation should
inform how scores are used.
- When local data can't establish linearity/monotonicity, consult past research and its
implications (e.g., cognitive ability tends to relate linearly to performance; linearity for other
predictors such as personality is less settled).
- When setting any cutoff, consider the conditional standard error of measurement at the cutoff
region; document the SEM model used. Consider reporting the percentage of applicants classified the
same way (pass/fail) across replications at the cutoff (Haertel, 2006).
Bands
Bands are score ranges within which candidates are treated alike (a form of cutoff that defines
ranges). Methods vary (Cascio, Outtz, Zedeck, & Goldstein, 1991; Campion et al., 2001).
- Rationale can be psychometric (imprecision/SEM of scores) and/or administrative/
organizational.
- Tradeoff: because banded candidates with different scores are treated alike, banding generally
yields lower expected criterion performance and utility than top-down selection — but may be
balanced by administrative ease and possibly increased workforce diversity, depending on how
within-band selection is done.
- Document the basis for development and the decision rules for administering the band.
Top-down vs. cutoff vs. bands — the driving factors
Decisions are typically driven by organizational goals and factors such as: estimated cost–benefit
ratio, number of vacancies, selection ratio, labor market, expectancy of success vs. failure,
consequences of selection errors, relative emphasis on performance vs. diversity goals, judgments
about the level of KSAO/performance required, and the procedure's utility. Some organizations choose a
cutoff over rank order to increase diversity, accepting possible reductions in performance and
utility. Whatever the decision, document the rationale.
Norms
Present normative information for the applicant pool and incumbent population when appropriate.
Describe the normative group's relevant demographic/occupational characteristics and the time
frame. Note that large discrepancies between incumbents and the applicant pool can make
incumbent-based cutoffs too high (or otherwise inappropriate).
Communicating effectiveness
- Expectancy charts relate score ranges to work performance; Taylor–Russell tables show the
proportion of hires who will be successful under combinations of validity, selection ratio, and
base rate.
- Utility estimates project productivity gains (in dollars, output %, or reductions in
accidents/person-hours). Utility values rest on assumptions and uncertain parameters — report
them as estimates, and present minimal and maximal point estimates to reflect uncertainty.
Appropriate use
Use a procedure only for purposes with validity evidence. Changing the mix or combination of
components (especially in a compensatory system) can fundamentally change the supported inference —
the original validation evidence may no longer suffice. Don't repurpose a procedure (e.g., diagnostic
use, or an education-designed test for employment) without supporting evidence.
Pitfalls
- Combining procedures without validating the combination, not just the parts.
- Setting cutoffs without considering linearity/monotonicity or the conditional SEM.
- Presenting utility as precise dollar truth rather than an assumption-laden estimate.
- Using incumbent norms to set cutoffs when the applicant pool differs markedly.
- Silently changing weights/combination and assuming prior validity carries over.
Checklist
See also
criterion-related-validation (weights, composites, corrections) · fairness-and-bias-analysis
(subgroup tradeoffs; analyze the operational composite) · technical-validation-report ·
administration-documentation
Source: Principles (5th ed., 2018), "Operational Considerations → Data Analyses (Combining
procedures, Multiple hurdles vs. compensatory, Cutoff scores vs. rank orders, Bands, Norms) and
Communicating the Effectiveness / Appropriate Use of Selection Procedures."
