Publication Type

PhD Dissertation

Version

publishedVersion

Publication Date

6-2020

Abstract

Dynamic malware analysis schemes either run the target program as is in an isolated environment assisted by additional hardware facilities or modify it with instrumentation code statically or dynamically. The hardware-assisted schemes usually trap the target during its execution to a more privileged environment based on the available hardware events. The more privileged environment is not accessible by the untrusted kernel, thus this approach is often applied for transparent and secure kernel analysis. Nevertheless, the isolated environment induces a virtual address gap between the analyzer and the target, which hinders effective and efficient memory introspection and undermines the correctness of semantics extraction. Code instrumentation mixes the analyzer code together with the target, thus they share the same execution flow as well as the virtual address space at runtime. The instrumentation code has native access capabilities to the target’s virtual memory, which seamlessly introspects and controls the target. However, code instrumentation based schemes are inadequate to tackle malicious execution since the analysis can be detected, evaded, or even tampered with as noted in many recent works.

We securely bridge the virtual address gap by designing a system called the On-site Analysis Infrastructure(OASIS) based on hardware virtualization technology. OASIS features a one-way address space sharing: on the one hand, the analyzer, as an independent full-fledged application, runs in a fused virtual address space comprising both its own space and the target’s; on the other hand, the analyzer’s space is separated and isolated from the target which still runs within its unmodified address space. We also extend OASIS with a significant instrumentation technique which allows secure transitions between the analyzer and the target without precipitating any CPU mode/privilege switch. In total, OASIS offers three capabilities to the analyzer: to reference the target virtual memory in a native way with mapping consistency; to dynamically control and instrument the target execution; and to transparently receive unmodified host OS services. With these capabilities, the analyzer performs onsite analysis on a malicious user/kernel thread running in the guest VM.

We propose two new dynamic analysis models based on OASIS: Onsite Memory Analysis (OMA) and Execution Flow Instrumentation (EFI). In OMA, the analyzer examines the user/kernel thread’s live virtual memory without interposing on its execution. We developed four tools to demonstrate its capability. The first one is a virtual machine introspection tool which is up to 87 times faster than the state of the art. The second one delineates the target’s virtual memory layout without trusting any kernel objects. The third one captures the target’s system call events along with their parameters without intercepting its execution. The last one generates the control flow graph for Just-In-Time emitted code. In EFI, the analyzer is provisioned with two options to directly intercept the user/kernel thread execution and dynamically instrument it. Despite being securely and transparently isolated from the target, the analyzer introspects and controls it in the same native way as the instrumentation code. We have also conducted three case studies. The first one is a cross-space control flow tracer which shows OASIS based EFI has better performance than existing hardware trapping based schemes. The second one works in tandem with Google Syzkaller which demonstrates EFI’s agility in controlling and introspecting the target thread. The last one examines how a user-space program exploits the vulnerability in dynamically loaded kernel modules. EFI tools are well-suited for targeted and fine-grained analysis.

We have implemented a prototype of OASIS on an x86-64 platform and have rigorously evaluated it with various experiments including performance and security tests. OASIS and its tools remain transparent and effective against targets armed with anti-analysis techniques including packing.

Keywords

System security, hardware virtualization, code instrumentation, VM introspection

Degree Awarded

PhD in Information Systems

Discipline

Programming Languages and Compilers | Software Engineering

Supervisor(s)

DING, Xuhua

First Page

1

Last Page

121

Publisher

Singapore Management University

City or Country

Singapore

Copyright Owner and License

Author

Download

Share

COinS