Publication Type

Conference Proceeding Article

Version

acceptedVersion

Publication Date

5-2024

Abstract

Short-answer based questions have been used widely due to their effectiveness in assessing whether the desired learning outcomes have been attained by students. However, due to their open-ended nature, many different answers could be considered entirely or partially correct for the same question. In the context of computer science and software engineering courses where the enrolment has been increasing recently, manual grading of short-answer questions is a time-consuming and tedious process for instructors. In software engineering courses, assessments concern not just coding but many other aspects of software development such as system analysis, architecture design, software processes and operation methodologies such as Agile and DevOps. However, existing work in automatic grading/scoring of text-based answers in computing courses have been focusing more on coding-oriented questions. In this work, we consider the problem of autograding a broader range of short answers in software engineering courses. We propose an automated grading system incorporating both text embedding and completion approaches based on recently introduced pre-trained large language models (LLMs) such as GPT-3.5/4. We design and implement a web-based system so that students and instructors can easily leverage autograding for learning and teaching. Finally, we conduct an extensive evaluation of our automated grading approaches. We use a popular public dataset in the computing education domain and a new software engineering dataset of our own. The results demonstrate the effectiveness of our approach, and provide useful insights for further research in this area of AI-enabled education.

Keywords

automatic grading, embedding, large language models, short answers, software engineering courses

Discipline

Educational Assessment, Evaluation, and Research | Higher Education | Software Engineering

Research Areas

Software and Cyber-Physical Systems

Publication

2024 IEEE Global Engineering Education Conference (EDUCON): Kos Island, Greece, May 8-11: Proceedings

First Page

1

Last Page

10

ISBN

9798350394023

Identifier

10.1109/EDUCON60312.2024.10578839

Publisher

IEEE

City or Country

Piscataway, NJ

Embargo Period

9-4-2024

Copyright Owner and License

Authors

Additional URL

https://doi.org/10.1109/EDUCON60312.2024.10578839

Download

Share

COinS