ChatGPT, can you generate multiple-choice questions for an undergraduate computing course? An evaluation of its effectiveness using classical test theory

Author

• Eng Lieh OUH, Singapore Management UniversityFollow
• Kar Way TAN, Singapore Management UniversityFollow
• Siaw Ling LO, Singapore Management UniversityFollow
• Benjamin GAN, Singapore Management UniversityFollow
• Yuhao ZHANG

Publication Type

Journal Article

Version

acceptedVersion

Publication Date

10-2026

Abstract

Educators are increasingly adopting large language models such as ChatGPT to generate multiple-choice questions (MCQs), yet empirical evidence on the quality of these items in computing education assessments remains limited. This study compares 30 ChatGPT-generated and 30 manually crafted items from an undergraduate software design course, and evaluates item difficulty, discrimination, and distractor quality using Classical Test Theory. We triangulated two evaluations using within-subjects and between-subjects designs. Our quantitative findings show that all three metrics in both evaluations point in the same direction. ChatGPT-generated items are easier, less discriminating, and exhibit poorer distractor quality than manually crafted items. Distractor quality of ChatGPT-generated items is significantly worse in both designs. Difficulty metric is significant in the within-subjects design and discrimination metric is significant in the between-subjects design. We discuss the quantitative findings for each evaluation and across the two evaluations. A qualitative analysis further identifies six construction patterns in ChatGPT-generated MCQs that contribute to non-functioning distractors and help explain their psychometric characteristics. These patterns provide practical guidance for instructors to review and revise ChatGPT-generated items before deployment.

Keywords

Classical Test Theory, multiple-choice questions, item analysis, point-biserial correlation, AI-assisted assessment, computing education

Discipline

Artificial Intelligence and Robotics | Instructional Media Design

Research Areas

Data Science and Engineering; Software and Cyber-Physical Systems

Areas of Excellence

Digital transformation

Publication

ACM Transaction on Computing Education

First Page

1

Last Page

26

Identifier

10.1145/3858331

Citation

OUH, Eng Lieh; TAN, Kar Way; LO, Siaw Ling; GAN, Benjamin; and ZHANG, Yuhao. ChatGPT, can you generate multiple-choice questions for an undergraduate computing course? An evaluation of its effectiveness using classical test theory. (2026). ACM Transaction on Computing Education. 1-26.
Available at: https://ink.library.smu.edu.sg/sis_research/11361

Copyright Owner and License

ACM

Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial-No Derivative Works 4.0 International License.

Additional URL

https://doi.org/10.1145/3858331