Publication Type

Journal Article

Version

acceptedVersion

Publication Date

2-2018

Abstract

In recent years, situational judgment tests (SJTs) have made strong inroads in assessment practices. Despite the importance of scoring for the validity of SJTs, little attention has been paid to different SJT scoring methods. This study investigated the influence of scoring methods on the criterion-related validity of SJTs. We examined five different consensus scoring methods (i.e., raw, standardized, dichotomous, mode, and proportion scoring) and several integrated scoring methods for scoring the same SJT. Results showed that one of the most popular scoring approaches (raw consensus scoring) is associated with an extreme response tendency and yields the lowest scale validity of all scoring approaches examined. Moreover, the mean item validity of midrange items was good only when they were scored by the mode consensus method. Thus, this study extends previous work (McDaniel et al., 2011) by deepening our understanding of how different scoring methods improve the validities of SJTs. Our findings suggest that using scoring methods that control the influence of extreme response tendency on the scores of SJTs yields higher validities. Finally, this study is the first to suggest that scoring SJTs with integrated methods yielded higher mean item validities than using any single method.

Keywords

Assessment, Situational judgment test, Extreme response tendency, Scoring method, Criterion-related validity

Discipline

Industrial and Organizational Psychology | Organizational Behavior and Theory

Research Areas

Organisational Behaviour and Human Resources

Publication

Journal of Vocational Behavior

Volume

104

First Page

199

Last Page

209

ISSN

0001-8791

Identifier

10.1016/j.jvb.2017.11.005

Publisher

Elsevier

Copyright Owner and License

Authors

Additional URL

https://doi.org/10.1016/j.jvb.2017.11.005

Download

Share

COinS