Conventional methods for assessing the validity and reliability of situational judgment test (SJT) scores have proven to be inadequate. For example, factor analysis techniques typically lead to nonsensical solutions, and assumptions underlying Cronbach's alpha coefficient are violated due to the multidimensional nature of SJTs. In the current article, we describe how cognitive diagnosis models (CDMs) provide a new approach that not only overcomes these limitations but that also offers extra advantages for scoring and better understanding SJTs. The analysis of the Q-matrix specification, model fit, and model parameter estimates provide a greater wealth of information than traditional procedures do. Our proposal is illustrated using data taken from a 23-item SJT that presents situations about student-related issues. Results show that CDMs are useful tools for scoring tests, like SJTs, in which multiple knowledge, skills, abilities, and other characteristics are required to correctly answer the items. SJT classifications were reliable and significantly related to theoretically relevant variables. We conclude that CDM might help toward the exploration of the nature of the constructs underlying SJT, one of the principal challenges in SJT research.
Situational judgment tests, cognitive diagnosis models, validity, reliability
Human Resources Management | Organizational Behavior and Theory
Organisational Behaviour and Human Resources
Organizational Research Methods
SAGE Publications (UK and US)
SORREL, Miguel A.; OLEA, Julio; ABAD, Francisco J.; DE LA TORRE, Jimmy; AGUADO, David; and LIEVENS, Filip.
Validity and reliability of situational judgement test scores: A new approach based on cognitive diagnosis models. (2016). Organizational Research Methods. 19, (3), 506-532. Research Collection Lee Kong Chian School Of Business.
Available at: http://ink.library.smu.edu.sg/lkcsb_research/5728
Creative Commons License
This work is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 4.0 License.