GameRTS: A regression testing framework for video games
Abstract
Continuous game quality assurance is of great importance to satisfy the increasing demands of users. To respond to game issues reported by users timely, game com-panies often create and maintain a large number of releases, updates, and tweaks in a short time. Regression testing is an essential technique adopted to detect regression issues during the evolution of the game software. However, due to the special characteristics of game software (e.g., frequent updates and long-running tests), traditional regression testing techniques are not directly applicable. To bridge this gap, in this paper, we perform an early exploratory study to investigate the challenges in regression testing of video games. We first performed empirical studies to better understand the game development process, bugs introduced during game evolution, and the context sensitivity. Based on the results of the study, we proposed the first regression test selection (RTS) technique for game software, which is a compromise between safety and practicality. In particular, we model the test suite of game software as a State Transition Graph (STG) and then perform the RTS on the STG. We establish the dependencies between the states/actions of STG and game files, including game art resources, game design files, and source code, and perform change impact analysis to identify the states/actions (in the STG) that potentially execute such changes. We implemented our framework in a tool, named GameRTS, and evaluated its usefulness on 10 tasks of a large-scale commercial game, including a total of 1,429 commits over three versions. The experimental results demonstrate the usefulness and effectiveness of GameRTS in game RTS. For most tasks, GameRTS only selected one trace from STG, which can significantly reduce the testing time. Furthermore, GameRTS detects all the regression bugs from the test evaluation suites. Compared with the file-level RTS, GameRTS selected fewer states/actions/traces (i.e., 13.77%, 23.97%, 6.85%). In addition, GameRTS identified 2 new critical regression bugs in the game.