Publication Type

PhD Dissertation

Version

publishedVersion

Publication Date

9-2025

Abstract

Software systems written by humans tend to be unreliable and insecure, hence, bugs or vulnerabilities in them are inevitable. Symbolic execution has shown considerable potential in detecting diverse types of software bugs and also vulnerabilities that have severe security implications. However, existing symbolic execution engines still suffer from at least three fundamental limitations in memory modeling, path exploration, and structured input generation, which significantly impede existing engines from efficiently and effectively detecting software bugs and vulnerabilities.

The objective of this dissertation is to boost existing symbolic execution engines by designing a new memory model, two new path exploration strategies, and a new test input generation solution to alleviate three key limitations and facilitate automatic vulnerability detection. Specifically, in the first work, we propose SymLoc, a vulnerability detection system that designs a new symbolic memory model. In the second work, we propose FastKLEE, a faster path exploration solution achieved by reducing redundant bound checking during execution. In the third work, we propose Vital to perform vulnerability-oriented path exploration for effective vulnerability detection. In the fourth work, we propose Cottontail, a LLM-driven concolic execution engine that could effectively generate highly structured test inputs for detecting vulnerabilities in parsing test programs.

The prototypes implemented in the dissertation have found many unknown vulnerabilities (e.g., buffer overflow and memory leakage) in widely used software systems, some of which are assigned with CVE (e.g., CVE-2024-55061).

Keywords

Software security, program analysis, symbolic execution, white-box fuzzing

Degree Awarded

PhD in Computer Science

Discipline

Information Security | Software Engineering

Supervisor(s)

JIANG, Lingxiao

First Page

1

Last Page

190

Publisher

Singapore Management University

City or Country

Singapore

Copyright Owner and License

Author

Download

Share

COinS