Generalizing Validity Evidence

Generalizing validity evidence

When accumulated evidence is strong enough, you may justify a procedure in a new setting without a local criterion study — by demonstrating generalized validity and making a compelling argument for direct applicability to your situation. Three strategies (not mutually exclusive, not exhaustive): transportability, synthetic/job-component validity, and meta-analytic validity generalization. All require an analysis of the work to establish the link.

1. Transportability

Apply a specific procedure in a new situation based on a validation study conducted elsewhere, when key similarities make that evidence applicable.

• Carefully review the original study for technical soundness and conceptual/empirical relevance to the new situation.
• Compare on job content, job requirements, job context, and (if feasible) the applicant group.
• Document the comparison between the original validation sample/setting and your target use.

2. Synthetic validity / job-component validity

Justify use based on the validity of inferences for one or more components (job components) of the work, established for those components, then "synthesized" (empirically combined) for a given job or job family.

• Requires a detailed analysis of work that decomposes jobs into components and identifies which components a job comprises.
• Powerful when many jobs share common components — provides a validity source where a separate criterion study per job isn't feasible, and reduces burdensome data collection.
• Caveat: job requirements unique to a job may not be well covered by components and may need other evidence.

3. Meta-analytic validity generalization (VG)

Cumulate validity findings across studies to estimate the predictor–criterion relationship for the domains/settings of interest, and to determine how specific/generalizable the relationship is.

• VG shows that much of the observed variation in validity across settings is due to statistical artifacts (sampling error, range restriction, criterion unreliability). Well established for cognitive ability; accruing for several noncognitive measures.
• Organize around constructs. Generalization is straightforward when results are organized by predictor and criterion constructs. Method labels alone (e.g., "interview," "SJT," "biodata") are not constructs — a method can be designed to assess very different constructs, so you can only generalize to applications of the method that share the relevant features (content, scoring, meaning of scores).
• Evaluate the meta-analysis itself: methods and assumptions, their tenability, statistical artifacts that could bias results, and moderators. When a substantive moderator is plausible, consider statistical power and the precision of reported effects to detect it. Insist on full reporting of how studies were categorized and analyzed; missing/unreported information undermines the inference.

Local study vs. generalized evidence

Generalized evidence is often more useful than a single small local study. But a competent local study with a large, organizationally relevant sample using the same test for the same work can be more accurate and informative than an accumulation of small, heterogeneous, or deficient studies that don't represent your setting. Watch for representation gaps: if your setting (e.g., a managerial job) isn't represented in the meta-analytic database (e.g., limited to entry-level jobs), a local study may be more relevant. A Bayesian approach can formally combine meta-analytic priors with locally estimated coefficients (Newman, Jacobs, & Bartram, 2007).

Sole reliance on cumulative evidence may also be insufficient for operational needs (e.g., optimal placement, combining procedures in a broader system) — supplementary local or cooperative studies may still be warranted.

When generalization is NOT warranted

• Generalizing from a method studied for one construct to the same method used for a different construct (e.g., technical-knowledge interviews → interpersonal-skill interviews).
• Applying meta-analytic results from one set of procedures/settings to a new setting using different, unspecified procedures.
• Assuming predictor/criterion measures sharing a construct label are interchangeable without rational and empirical support.

Pitfalls

• Skipping the analysis of work that links the borrowed evidence to your job(s) — required for all three strategies.
• Organizing meta-analytic evidence by method label ("interview," "SJT") instead of by construct.
• Treating generalized evidence as automatically superior, ignoring representation gaps (your setting absent from the meta-analytic database) where a local study would be more relevant.
• Transporting a study without reviewing its technical soundness and comparing job content, requirements, context, and applicant group.
• Relying solely on cumulative evidence when operational needs (placement, combining procedures) require local data.

Checklist

• Strategy chosen (transport / synthetic / VG) with rationale
• Analysis of work links the borrowed evidence to your job(s)
• Transport: original study reviewed; content/requirements/context/applicant comparison documented
• Synthetic: job decomposed into components; component validities established
• VG: evidence organized by constructs, not method labels
• Meta-analysis methods, assumptions, artifacts, and moderators evaluated
• Representation gaps and operational-need gaps assessed; local/Bayesian supplement considered
• Direct-applicability argument to the current setting articulated

See also

validation-planning · work-analysis (required link) · criterion-related-validation (local alternative/supplement) · fairness-and-bias-analysis · technical-validation-report

Source: Principles (5th ed., 2018), "Generalizing Validity Evidence."