Publication Type

Conference Proceeding Article

Version

publishedVersion

Publication Date

7-2026

Abstract

This paper introduces a Knowledge‑State Generative Agent framework for evaluating the quality of pre‑assessment questions. The framework employs large language model (LLM)–based agents prompted to adopt a teacher persona to simulate the responses of students with and without mastery of targeted knowledge components. A preliminary empirical study using archival data from 424 students enrolled in an Information Systems Management course indicates that the proposed approach yields interpretable metrics under Classical Test Theory. Results further show that agents instantiated with the relevant mastered knowledge components exhibit systematically higher performance than agents lacking such mastery. In addition, the study suggests that teacher-persona prompting may mitigate the high-accuracy bias observed in LLMs, whereby models tend to produce correct answers even when prompted to disregard latent knowledge acquired during training. Future work will extend the framework beyond multiple-choice questions and evaluate its generalizability across interdisciplinary domains.

Keywords

Large Language Model, Classical Test Theory, Knowledge Component, High-accuracy Bias

Discipline

Artificial Intelligence and Robotics

Research Areas

Information Systems and Management

Areas of Excellence

Digital transformation

Publication

Proceedings of the 2026 Pacific Asia Conference on Information Systems (PACIS), Jakarta, Indonesia, July 4-8

First Page

1

Last Page

9

Publisher

AIS

City or Country

Jakarta, Indonesia

Download

Share

COinS