Publication Type

Journal Article

Version

acceptedVersion

Publication Date

3-2023

Abstract

Recently, multiple, speeded assessments (e.g., “speeded” or “flash” role-plays) have made rapid inroads into the selection domain. So far, however, the conceptual underpinning and empirical evidence related to these short, fast-paced assessment approaches has been lacking. This raises questions whether these speeded assessments can serve as reliable and valid indicators of future performance. This paper uses the notions of stimulus and response domain sampling to conceptualize multiple, speeded behavioral job simulations as a hybrid of established simulation-based selection methods. Next, we draw upon the thin slices of behavior paradigm to theorize about the quality of ratings made in multiple, speeded behavioral simulations. In two studies, various assessor pools assessed a sample of 96 MBA students in eighteen 3-minute role-plays designed to capture situations in the junior management domain. At the individual speeded role-play level, reliability and validity were not ensured. Yet, aggregated across all assessors’ ratings of all speeded role-plays, the overall score for predicting future performance was high (.54). Validities remained high when assessors evaluated only the first minute (vs. full 3 minutes) or received only a control training (vs. traditional assessor training). Aggregating ratings of performance in multiple, heterogeneous situations that elicit a variety of domain-relevant behavior emerged as key requirement to obtain adequate domain coverage, capture both ability and personality (extraversion and agreeableness), and achieve substantial validities. Overall, these results show the importance of the stimulus and response domain sampling logic and send a strong warning to using “single” speeded behavioral simulations in practice.

Keywords

personnel selection, multiple, speeded assessments, behavioral job simulations, thin slices, generalizability theory, stimulus domain and response domain sampling

Discipline

Industrial and Organizational Psychology | Organizational Behavior and Theory

Research Areas

Organisational Behaviour and Human Resources

Publication

Journal of Applied Psychology

Volume

108

Issue

3

First Page

351

Last Page

373

ISSN

0021-9010

Identifier

10.1037/apl0000603

Publisher

American Psychological Association

Embargo Period

6-21-2022

Copyright Owner and License

Authors

Additional Files
APL-2020-0369_Supplemental_Materials.docx (100 kB)
Supplementary data

Additional URL

https://doi.org/10.1037/apl0000603

Download

Download Data/Files

Share

COinS