Publication Type

Journal Article

Version

publishedVersion

Publication Date

12-2025

Abstract

Short-answer questions are commonly used in educational assessments, as they are often viewed as a more effective way than multiple-choice questions to determine whether students have achieved the intended learning outcomes. However, manually creating appropriate questions targeting different cognitive levels such as those defined by the Bloom’s Taxonomy, and grading text answers from students are not trivial tasks for instructors. Existing work on auto-question generation and scoring in computing education typically targets coding-based questions. However, in software engineering courses, assessments can extend beyond coding to understanding of processes, DevOps methodologies, system design, etc. This work aims to address the dual problem of auto-generating and auto-scoring text based short-answer questions in software engineering courses. We propose an agent-based approach to question generation and scoring, in which each agent performs a distinct task and communicates with each other to complete its tasks. An agent can be realized using a selected large language model (LLM). To this end, we have implemented the proposed approach into a fully functional web app, utilizing OpenAI’s GPT-3.5 and GPT-4o models, which was deployed to support instructors and students in our course. We conducted a thorough evaluation of the agent-based approach using several public and private datasets. The results demonstrated the effectiveness of our approach. Among other insights, we observed that the LLM agents are good at generating questions at Bloom’s cognitive levels of Remember, Understand, Apply, and Analyze; and are good at scoring students’ answers when provided with 3 or more actual examples of human scoring.

Keywords

Agent based systems, large language models, short-answer questions, auto-scoring, question generation

Discipline

Artificial Intelligence and Robotics | Software Engineering

Research Areas

Intelligent Systems and Optimization

Areas of Excellence

Digital transformation

Publication

IEEE Access

Volume

13

First Page

215804

Last Page

215821

ISSN

2169-3536

Identifier

10.1109/ACCESS.2025.3647111

Publisher

Institute of Electrical and Electronics Engineers

Additional URL

https://doi.org/10.1109/ACCESS.2025.3647111

Download

Share

COinS