Publication Type

Conference Proceeding Article

Version

acceptedVersion

Publication Date

10-2024

Abstract

Large Language Models (LLMs) have taken the world by storm, demonstrating their ability not only to automate tedious tasks, but also to show some degree of proficiency in completing software engineering tasks. A key concern with LLMs is their “black-box” nature, which obscures their internal workings and could lead to societal biases in their outputs. In the software engineering context, in this early results paper, we empirically explore how well LLMs can automate recruitment tasks for a geographically diverse software team. We use OpenAI's ChatGPT to conduct an initial set of experiments using GitHub User Profiles from four regions to recruit a six-person software development team, analyzing a total of 3,657 profiles over a five-year period (2019–2023). Results indicate that ChatGPT shows preference for some regions over others, even when swapping the location strings of two profiles (counterfactuals). Furthermore, ChatGPT was more likely to assign certain developer roles to users from a specific country, revealing an implicit bias. Overall, this study reveals insights into the inner workings of LLMs and has implications for mitigating such societal biases in these models.

Keywords

Large Language Models, GitHub, Open Source, Software, Software Team Recruitment

Discipline

Software Engineering

Research Areas

Intelligent Systems and Optimization

Areas of Excellence

Digital transformation

Publication

Proceedings of the 2024 IEEE International Conference on Software Maintenance and Evolution (ICSME), Flagstaff, AZ, USA, October 6-11

First Page

624

Last Page

629

Identifier

10.1109/ICSME58944.2024.00063

Publisher

IEEE Computer Society

City or Country

Los Alamitos, CA

Additional URL

https://doi.ieeecomputersociety.org/10.1109/ICSME58944.2024.00063

Download

Share

COinS