Analyzing and revivifying function signature inference using deep learning
Publication Type
Journal Article
Publication Date
5-2024
Abstract
Function signature plays an important role in binary analysis and security enhancement, with typical examples in bug finding and control-flow integrity enforcement. However, recovery of function signatures by static binary analysis is challenging since crucial information vital for such recovery is stripped off during compilation. Although function signature recovery using deep learning (DL) is proposed in an effort to handle such challenges, the reported accuracy is low for binaries compiled with optimizations. In this paper, we first perform a systematic study to quantify the extent to which compiler optimizations (negatively) impact the accuracy of existing DL techniques based on Recurrent Neural Network (RNN) for function signature recovery. Our experiments show that the state-of-the-art DL technique has its accuracy dropped from 98.7% to 87.7% when training and testing optimized binaries. We further investigate the type of instructions that existing RNN model deems most important in inferring function signatures with the help of saliency map. The results show that existing RNN model mistakenly considers non-argument-accessing instructions to infer the number of arguments, especially when dealing with optimized binaries. Finally, we identify specific weaknesses in such existing approaches and propose an enhanced DL approach named ReSIL to incorporate compiler-optimization-specific domain knowledge into the learning process. Our experimental results show that ReSIL significantly improves the accuracy and F1 score in inferring function signatures, e.g., with accuracy in inferring the number of arguments for callees compiled with optimization flag O1 from 84.83% to 92.68%. Meanwhile, ReSIL correctly considers the argument-accessing instructions as the most important ones to perform the inferencing. We also demonstrate security implications of ReSIL in Control-Flow Integrity enforcement in stopping potential Counterfeit Object-Oriented Programming (COOP) attacks.
Keywords
Function signature, recurrent neural network, compiler optimization, control-flow integrity
Discipline
Software Engineering
Research Areas
Software and Cyber-Physical Systems
Publication
Empirical Software Engineering
Volume
29
Issue
69
First Page
1
Last Page
48
ISSN
1382-3256
Identifier
10.1007/s10664-024-10453-9
Publisher
Springer
Citation
LIN, Yan; SINGHAL, Trisha; GAO, Debin; and LO, David.
Analyzing and revivifying function signature inference using deep learning. (2024). Empirical Software Engineering. 29, (69), 1-48.
Available at: https://ink.library.smu.edu.sg/sis_research/9262
Additional URL
https://doi.org/10.1007/s10664-024-10453-9