"Human-AI synergy in survey development: Implications from Large Langua" by Ping Fan KE and Ka Chung NG
 

Publication Type

Journal Article

Version

publishedVersion

Publication Date

2-2025

Abstract

This study examines the novel integration of Large Language Models (LLMs) into the survey development process in business and research through the development and evaluation of the Behavioral Research ASSistant (BRASS) Bot. We first analyzed the traditional scale development process to identify tasks suitable for LLM integration, including both human-in-the-loop and automated LLM data collection methods. Following this analysis, we developed the details of BRASS Bot, incorporating design principles of falsifiability and reproducibility. We then conducted a comprehensive evaluation of the BRASS Bot across a diverse set of LLMs, including GPT, Claude, Gemini, and Llama, to assess its usability, validity, and reliability. We further demonstrated the practical utility of the BRASS Bot by conducting a user study and a predictive validity simulation. Our research presents both theoretical and practical implications. The augmentation approach of the BRASS Bot enriches the theoretical foundations of behavioral constructs byidentifying previously overlooked patterns. Additionally, the BRASS Bot offers significant time and resource efficiency gains while enhancing scale validity. Our work lays the foundation for future research on the broader application of LLMs as both assistants and collaborators in survey analysis and behavioral research design and execution, highlighting their potential for a transformative impact on the field.

Keywords

Large Language Model, Generative AI, Scale Development, Behavioral Research

Discipline

Artificial Intelligence and Robotics | Computer Sciences | Databases and Information Systems

Research Areas

Data Science and Engineering

Publication

ACM Transactions on Management Information Systems

Volume

16

Issue

1

First Page

1

Last Page

47

ISSN

2158-656X

Identifier

10.1145/3700597

Publisher

Association for Computing Machinery (ACM)

Copyright Owner and License

Authors

Creative Commons License

Creative Commons Attribution 4.0 International License
This work is licensed under a Creative Commons Attribution 4.0 International License.

Additional URL

https://doi.org/10.1145/3700597

Plum Print visual indicator of research metrics
PlumX Metrics
  • Usage
    • Downloads: 6
    • Abstract Views: 1
  • Captures
    • Readers: 16
  • Mentions
    • News Mentions: 1
see details

Share

COinS