Publication Type

Journal Article

Version

acceptedVersion

Publication Date

10-2024

Abstract

This study examines the novel integration of Large Language Models (LLMs) into the survey development process in business and research through the development and evaluation of the Behavioral Research ASSistant (BRASS) Bot. We first analyzed the traditional scale development process to identify tasks suitable for LLM integration, including both human-in-the-loop and automated LLM data collection methods. Following this analysis, we developed the details of BRASS Bot, incorporating design principles of falsifiability and reproducibility. We then conducted a comprehensive evaluation of the BRASS Bot across a diverse set of LLMs, including GPT, Claude, Gemini, and Llama, to assess its usability, validity, and reliability. We further demonstrated the practical utility of the BRASS Bot by conducting a user study and a predictive validity simulation. Our research presents both theoretical and practical implications. The augmentation approach of the BRASS Bot enriches the theoretical foundations of behavioral constructs byidentifying previously overlooked patterns. Additionally, the BRASS Bot offers significant time and resource efficiency gains while enhancing scale validity. Our work lays the foundation for future research on the broader application of LLMs as both assistants and collaborators in survey analysis and behavioral research design and execution, highlighting their potential for a transformative impact on the field.

Keywords

Large Language Model, Generative AI, Scale Development, Behavioral Research

Discipline

Computer Sciences | Databases and Information Systems

Research Areas

Data Science and Engineering; Information Systems and Management

Publication

ACM Transactions on Management Information Systems

ISSN

2158-656X

Identifier

10.1145/3700597

Publisher

Association for Computing Machinery (ACM)

Copyright Owner and License

ACM

Download

Share

COinS