Publication Type

Conference Proceeding Article

Version

publishedVersion

Publication Date

4-2024

Abstract

Identifying the misconceptions of novice programmers is pertinent for informing instructors of the challenges faced by their students in learning computer programming. In the current literature, custom tools, test scripts were developed and, in most cases, manual effort to go through the individual codes were required to identify and categorize the errors latent within the students' code submissions. This entails investment of substantial effort and time from the instructors. In this study, we thus propose the use of ChatGPT in identifying and categorizing the errors. Using prompts that were seeded only with the student's code and the model code solution for questions from two lab tests, we were able to leverage on ChatGPT's natural language processing and knowledge representation capabilities to automatically collate frequencies of occurrence of the errors by error types. We then clustered the generated error descriptions for further insights into the misconceptions of the students. The results showed that although ChatGPT was not able to identify the errors perfectly, the achieved accuracy of 93.3% is sufficiently high for instructors to have an aggregated picture of the common errors of their students. To conclude, we have proposed a method for instructors to automatically collate the errors latent within the students' code submissions using ChatGPT. Notably, with the novel use of generated error descriptions, the instructors were able to have a more granular view of the misconceptions of their students, without the onerous effort of manually going through the students' codes.

Keywords

LLM, ChatGPT, misconception, programming, errors, cluster, prompts

Discipline

Artificial Intelligence and Robotics | Software Engineering

Research Areas

Data Science and Engineering

Publication

ICSE-SEET '24: Proceedings of the 46th IEEE/ACM International Conference on Software Engineering: Software Engineering Education and Training: Portugal, April 14-20

First Page

233

Last Page

241

ISBN

9798400704987

Identifier

10.1145/3639474.3640059

Publisher

ACM

City or Country

New York

Additional URL

https://doi.org/doi/10.1145/3639474.3640059

Download

Share

COinS