Publication Type
Journal Article
Version
publishedVersion
Publication Date
2-2024
Abstract
Source-to-source code translation automatically translates a program from one programming language to another. The existing research on code translation evaluates the effectiveness of their approaches by using either syntactic similarities (e.g., BLEU score), or test execution results. The former does not consider semantics, the latter considers semantics but falls short on the problem of insufficient data and tests. In this paper, we propose MBTA (Mutation-based Code Translation Analysis), a novel application of mutation analysis for code translation assessment. We also introduce MTS (Mutation-based Translation Score), a measure to compute the level of trustworthiness of a translator. If a mutant of an input program shows different test execution results from its translated version, the mutant is killed and a translation bug is revealed. Fewer killed mutants indicate better code translation. MBTA is novel in the sense that mutants are compared to their translated counterparts, and not to their original program’s translation. We conduct a proof-of-concept case study with 612 Java-Python program pairs and 75,082 mutants on the code translators TransCoder and j2py to evaluate the feasibility of MBTA. The results reveal that TransCoder and j2py fail to translate 70.44% and 70.64% of the mutants, respectively, i.e., more than two-thirds of all mutants are incorrectly translated by these translators. By analysing the MTS results more closely, we were able to reveal translation bugs not captured by the conventional comparison between the original and translated programs.
Keywords
Mutation testing, Source to source translation, Code translation
Discipline
Software Engineering
Research Areas
Software and Cyber-Physical Systems
Publication
Empirical Software Engineering
Volume
29
Issue
1
First Page
1
Last Page
23
ISSN
1382-3256
Identifier
10.1007/s10664-023-10385-w
Publisher
Springer
Citation
GUIZZO, Giovani; ZHANG, Jie M.; SARRO, Federica; TREUDE, Christoph; and HARMAN, Mark.
Mutation analysis for evaluating code translation. (2024). Empirical Software Engineering. 29, (1), 1-23.
Available at: https://ink.library.smu.edu.sg/sis_research/8780
Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-No Derivative Works 4.0 International License.
Additional URL
https://doi.org/10.1007/s10664-023-10385-w