Meno:Vladislav
Priezvisko:Hrčka
Názov:Analysis of Virtual Machine Based Obfuscators
Vedúci:RNDr. Jaroslav Janáček, PhD
Rok:2023
Kµúčové slová:virtual-machine reverse-engineering obfuscation symbolic-execution
Abstrakt:Code Virtualizer is a powerful code obfuscation system for Windows applications that helps developers to protect sensitive code areas against Reverse Engineering with very strong obfuscation code, based on code virtualization. Apart from virtual machines, the obfuscation system additionally introduces several other obfuscation techniques such as insertion of junk code, encoding of virtual operands, duplication of virtual opcodes, opaque predicates, merging of virtual instructions, and another nested virtual machine. Code Virtualizer's virtual machines, with rest of the obfuscation techniques, have been increasingly being used for illicit purposes such as protection of malware. In this thesis we analyze the internals of a Code Virtualizer's virtual machine used in a malware sample and describe our semiautomatic approach to seeing through the obfuscation techniques in reasonable time. We demonstrate the approach on a few chunks of bytecode of the protected malware sample and compare the results against a non-obfuscated sample to confirm the validity of the method. Our approach is based on a known deobfuscation method that extracts the semantics of the virtual opcodes, using symbolic execution with simplifying rules. We further treat the bytecode chunks and certain internal constructs of the virtual machine as concrete values instead as symbolic ones, which enables the known deobfuscation method to deal with the additional obfuscation techniques automatically. In summary, we analyze the underlying parts of the virtual machine and gradually shape our deobfuscation approach. We describe our implementation of the approach at the end of the thesis.

Súbory diplomovej práce:

wslink-vm-analyzer-master.zip
dp_hrcka_final.pdf

Súbory prezentácie na obhajobe:

hrcka_dp.pptx
hrcka_dp(1).pdf

Upravi»