A short scale’s reliability and validity have to be shown again in the people you will infer about

What it is

Short scales are often compressed from longer instruments. Reliability, factor structure, and criterion correlations in the original paper belong to those people, those tasks, and that language. They do not travel with the items into a new product domain, age band, channel, or language. The short form has to have reliability and validity re-estimated in the target population. “The authors reported an acceptable alpha” is not evidence about the present sample. What migrates is item text, not psychometric properties.

Why it happens

Compression usually keeps items with the highest loadings or the highest item–total correlations in some normative sample. Those items are optimal for that sample and need not be optimal for others. Side weights of the construct shift in a new population: specialists care more about error recovery, newcomers more about first-time success; a short form that kept only the former loses validity for the latter. Translation, device, and timing (an in-task prompt versus a post-task email) also change what an item means. Coefficients in the original report have already been inflated by the compression; treating them as a floor in a new sample is systematically optimistic.

Studying it

Run a local validation in the target population: internal structure, retest or an alternate reliability, correlations with pre-committed criteria (task success, time, behavioral retention), and known-group differences. The sample must cover the people who will be inferred about, not convenient colleagues. If the form will compare products, test measurement invariance as well. Put the original paper’s coefficients in an appendix; the main text reports only local ones. On a failed validation, return to a longer instrument or shrink the claim to an exploratory signal.

Where it stops holding

In a highly homogeneous group that almost matches the original norm, local validation may confirm rather than overturn, and it should still be done because compression itself changes properties. One local validation also does not cover every future version and country of the product. If the criterion shares the same response bias as the scale, a high correlation is shared method, not validity. Tiny “validations” will treat noise as structure; acceptable precision has to be specified in advance.

Applying it

• Attach a local validation plan when introducing any short form; until it is done, the form is a pilot, not a target or an external benchmark.
• Method sections may not cite only the original paper’s alpha or correlations; current-sample coefficients must appear.
• After a change of population, language, or task type, mark the validation expired and rerun it.
• Check: if structure and criteria in the current sample miss the pre-registered bar, remove the short form from the primary-metric list.

Related

• Same group: Q3.16.1 Single-item error cannot be checked by internal consistency · Q3.16.2 Short scales trade dimensional resolution for lower burden · Q3.16.4 A single rating cannot locate which process step failed
• Adjacent: Q3.15 Questionnaires and rating scales · Q4.10 Generalizability of research claims
• Search terms: short-form validation · validity transport · measurement invariance