Internal Structure Validation

Internal-structure validation

Evidence about how the components of a procedure (items, tasks, subscales) relate to one another and conform to the conceptual framework that defines the procedure. Useful for planning and developing a procedure and for supporting interpretation of total and subscores.

The critical limitation — state it up front

Internal-structure evidence does not, by itself, establish job relatedness. It shows the procedure measures the intended construct(s); it does not show those scores relate to work behavior or outcomes. Job relatedness still requires linking scores to the work via criterion-related-validation, content-based-validation, or generalizing-validity-evidence. Treat internal structure as support, not a stand-alone case.

Match the analysis to the conceptual framework

The relevant analyses depend on the procedure's intended structure:

• Single dimension/construct intended → offer evidence that covariances among components are accounted for by a strong single factor. Coefficient alpha / internal consistency can be appropriate evidence here.
• Multidimensional structure intended (hypothesized multifactor measure) → carefully examine dimensionality (e.g., confirmatory factor analysis fitting the proposed structural model). Overall internal consistency may be inappropriate as the index of choice.

Item inclusion should be driven primarily by relevance to the construct/content domain and only secondarily by intercorrelations. Well-constructed components with near-zero correlations with other components, scales, or the total score should not automatically be eliminated — if the procedure deliberately spans different construct/content domains (e.g., a battery of a reading test, an in-basket, and an interview), low inter-component correlations are expected and fine.

Cautions on coefficient alpha

High internal consistency can be misleading:

• A long, multi-dimensional measure can show high alpha simply because of the number of items (Cortina, 1993), masking multidimensionality.
• A performance-rating form with theoretically unrelated scales can show high alpha because of halo effect, not true unidimensionality.

So a high alpha is not proof of a single construct. Generic internal-consistency indices do not evaluate internal structure for multidimensional procedures.

Subscores

When a multidimensional structure is proposed, evidence supporting inferences about subcomponent scores may be needed if those subscores are interpreted or used. Don't report/interpret subscores you haven't supported.

Process

1. State the conceptual framework: how many dimensions/constructs, and how components map to them.
2. Choose analyses that fit that framework (single-factor evidence vs. CFA/dimensionality study).
3. Select components by construct/content relevance first; don't purge low-correlating items that belong conceptually.
4. Use alpha only where a single dimension is intended; otherwise examine dimensionality directly and beware item-count and halo artifacts.
5. Support any subscores you intend to interpret.
6. Remember to establish job relatedness separately.

Pitfalls

• Treating internal-structure evidence as establishing job relatedness — it does not; that requires a separate source linking scores to work behavior/outcomes.
• Reporting a high coefficient alpha as proof of unidimensionality (item-count and halo artifacts).
• Purging well-constructed components that correlate near-zero with others when they belong conceptually to a deliberately multidimensional procedure.
• Using overall internal consistency as the index of choice for a hypothesized multifactor measure.
• Interpreting or reporting subscores without supporting evidence for those subcomponent inferences.

Checklist

• Intended dimensionality (uni- vs. multi-) stated before analysis
• Analysis matched to that framework (single-factor evidence or CFA)
• Items/components chosen on construct/content relevance, not just correlations
• Alpha used only for intended-unidimensional measures; artifacts considered
• Subscores supported if interpreted
• Job-relatedness evidence obtained via a separate source

See also

criterion-related-validation · content-based-validation · generalizing-validity-evidence · fairness-and-bias-analysis (DIF is an item-level structural concern) · technical-validation-report

Source: Principles (5th ed., 2018), "Sources of Validity Evidence → Evidence of Validity Based on Internal Structure."