Publication Type

Journal Article

Version

publishedVersion

Publication Date

2-2027

Abstract

With the advancement of AI models, more software systems are adopting AI as a component to facilitate automation. Pre-trained models (PTMs) have become a cornerstone of AI-based software, allowing for rapid integration and development with lower training cost. However, their adoption also introduces failure modes such as data leakage and biased outputs, that may require careful handling by downstream developers. While previous research has proposed taxonomies of these technical concerns and various mitigation strategies, how downstream developers address these issues during the development of general AI-based software when reusing PTMs remains unexplored. Understanding downstream developers’ perspectives is essential because they directly influence how these potential failure concerns translate into practice, such as determining whether immediate risks like data leakage or model bias are recognised, mitigated, or inadvertently overlooked in real-world deployments. This study investigates downstream developers’ concerns, practices and perceived challenges regarding practical AI failures during the development of AI-based software. To achieve this, we conducted a mixed-method study, including interviews with 16 participants, a survey of 86 practitioners, and an analysis of 874 AI incidents from the AI Incident Database. Our results reveal that while developers generally demonstrate strong awareness of potential AI failures, their practices, especially during the preparation and model selection phases, are often inadequate. The lack of concrete guidelines and policies leads to significant variability in the comprehensiveness of their approaches throughout the development lifecycle, with additional challenges such as poor documentation and knowledge gaps, further impeding effective implementation. Based on our findings, we offer suggestions for AI model contributors, developers of AI-based software, researchers, and policy makers to enhance the integration of failure mitigation measures aimed at mitigating direct harms from AI failures.

Keywords

AI Failure, AI-Based Software, Empirical Software Engineering, Mixed Methods, Pre-Trained Models

Discipline

Artificial Intelligence and Robotics | Software Engineering

Research Areas

Intelligent Systems and Optimization

Areas of Excellence

Digital transformation

Publication

Empirical Software Engineering

Volume

32

Issue

1

First Page

1

Last Page

41

ISSN

1382-3256

Identifier

10.1007/s10664-026-10937-w

Publisher

Springer

Additional URL

https://doi.org/10.1007/s10664-026-10937-w

Download

Share

COinS