Publication Type

Journal Article

Version

acceptedVersion

Publication Date

7-2016

Abstract

Conventional methods for assessing the validity and reliability of situational judgment test (SJT) scores have proven to be inadequate. For example, factor analysis techniques typically lead to nonsensical solutions, and assumptions underlying Cronbach's alpha coefficient are violated due to the multidimensional nature of SJTs. In the current article, we describe how cognitive diagnosis models (CDMs) provide a new approach that not only overcomes these limitations but that also offers extra advantages for scoring and better understanding SJTs. The analysis of the Q-matrix specification, model fit, and model parameter estimates provide a greater wealth of information than traditional procedures do. Our proposal is illustrated using data taken from a 23-item SJT that presents situations about student-related issues. Results show that CDMs are useful tools for scoring tests, like SJTs, in which multiple knowledge, skills, abilities, and other characteristics are required to correctly answer the items. SJT classifications were reliable and significantly related to theoretically relevant variables. We conclude that CDM might help toward the exploration of the nature of the constructs underlying SJT, one of the principal challenges in SJT research.

Keywords

Situational judgment tests, cognitive diagnosis models, validity, reliability

Discipline

Human Resources Management | Industrial and Organizational Psychology | Organizational Behavior and Theory

Research Areas

Organisational Behaviour and Human Resources

Publication

Organizational Research Methods

Volume

19

Issue

3

First Page

506

Last Page

532

ISSN

1094-4281

Identifier

10.1177/1094428116630065

Publisher

SAGE

Copyright Owner and License

Authors

Additional URL

https://doi.org/10.1177/1094428116630065

Download

Share

COinS