Malware Analysis Advanced

Malware Analysis Advanced

Summary

Advanced malware analysis is the discipline of unpacking, reverse engineering, and understanding modern packed/obfuscated malware (UPX, VMProtect, Themida, Enigma, custom packers), identifying sandbox-evasion techniques (anti-VM, anti-debug, anti-analysis), analyzing rootkits (user-mode, kernel-mode, bootkits, UEFI), and authoring detection rules (YARA). This domain covers full unpacking workflows (static + dynamic), modern threat actor tooling (Emotet, TrickBot, Conti, LockBit, BlackCat/ALPHV, REvil), YARA rule authoring and optimization, and industry-standard analyst tooling (IDA Pro, Ghidra, Binary Ninja, radare2). Distinct from foundational binary-reverse — this skill focuses on dynamic unpacking, evasion triage, rootkit techniques, and analyst workflow automation.

Key Terms

• Packer — Tool that compresses + encrypts executable to evade AV (UPX, VMProtect, Themida)
• Unpacking — Recovering original executable from packed binary
• Anti-VM — Code that detects virtual machine (sandbox evasion)
• Anti-debug — Code that detects debugger presence (analyst evasion)
• Anti-analysis — Umbrella term for AV/VM/debug/sandbox evasion
• Rootkit — Tool that hides processes / files / network connections
• User-mode rootkit — Rootkit running in ring 3 (DLL injection, API hooking)
• Kernel-mode rootkit — Rootkit running in ring 0 (driver, syscall hooking)
• Bootkit — Rootkit that infects bootloader (MBR / VBR)
• UEFI rootkit — Rootkit that infects UEFI firmware (persistent across reinstall)
• YARA — Pattern-matching tool for malware identification
• PE — Portable Executable format (Windows binaries)
• ELF — Executable and Linkable Format (Linux binaries)
• Triage — Initial malware assessment (severity, family, capability)
• Sandbox — Isolated analysis environment (Cuckoo, JoeSandbox, Any.Run)

Scope

This skill covers advanced malware analysis:

• Unpacking modern packers (UPX, VMProtect, Themida, Enigma, custom)
• Sandbox-evasion detection (anti-VM, anti-debug, anti-analysis)
• Rootkit analysis (user-mode, kernel-mode, bootkits, UEFI)
• YARA rule authoring and optimization
• IDA Pro / Ghidra / Binary Ninja workflows
• Modern threat tooling (Emotet, TrickBot, Conti, LockBit, BlackCat/ALPHV, REvil)
• Automated triage pipelines

Out of scope: foundational RE (see binary-reverse), exploit development (see exploit-development), reverse engineering theory (see reverse-engineering-advanced).

Use Cases

• Unpacking packed malware: Recover original code from VMProtect/Themida binaries
• Sandbox evasion triage: Identify anti-VM/anti-debug techniques
• Rootkit detection: Find user-mode/kernel-mode rootkit techniques
• YARA rule authoring: Detect malware families by signature
• Threat actor tooling analysis: Reverse engineer Emotet, TrickBot, Conti, LockBit, BlackCat, REvil
• Automated triage pipeline: Build Cuckoo/JoeSandbox pipeline for mass triage
• Detection rule optimization: Tune YARA rules for performance
• Memory forensics: Use Volatility for in-memory malware analysis
• UEFI analysis: Identify UEFI rootkits (LoJax, MosaicRegressor)
• API hook detection: Find user-mode rootkit hooks

Core Tools

Methodology

Phase 1 — Static triage

# File hash
sha256sum malware.exe

# File type
file malware.exe

# PE analysis
pe-tree malware.exe

# Strings
strings -a malware.exe | grep -iE "http|dll|reg|cmd"

# Section entropy (packed indicator)
python3 -c "
import pefile
pe = pefile.PE('malware.exe')
for section in pe.sections:
    print(f'{section.Name.decode().strip():12s} entropy={section.get_entropy():.2f}')
"

Phase 2 — Unpacking

# UPX
upx -d malware.exe -o malware_unpacked.exe

# VMProtect / Themida / custom
# Use x64dbg + Scylla to dump from memory
# 1. Load in x64dbg
# 2. Set breakpoint on OEP (original entry point)
# 3. Run until OEP hit
# 4. Use Scylla to dump process memory
# 5. Fix IAT (Import Address Table)

Phase 3 — Dynamic analysis

# Run in sandbox (Cuckoo)
cuckoo submit malware.exe

# Manual analysis with x64dbg
# 1. Load binary
# 2. Set breakpoints on WinAPI calls (CreateFile, WriteFile, etc.)
# 3. Run, observe behavior
# 4. Capture network traffic (Wireshark)

# Volatility memory analysis
volatility -f memory.dmp windows.pslist
volatility -f memory.dmp windows.netscan
volatility -f memory.dmp windows.malfind

Phase 4 — Sandbox evasion detection

# Static anti-VM strings
strings malware.exe | grep -iE "vmware|virtualbox|qemu|hyper-v|xen"
strings malware.exe | grep -iE "vbox|vmware tools|prl_"

# Anti-debug APIs
strings malware.exe | grep -iE "IsDebuggerPresent|CheckRemoteDebuggerPresent|NtQueryInformationProcess"

# Dynamic API tracing
# Use API monitor to capture all API calls

Phase 5 — Rootkit analysis

# User-mode rootkit detection
pe-sieve /pid 1234 /imp 3
hollows-hunter /pid 1234

# Kernel-mode rootkit detection
# Use WinDbg kernel mode
# List loaded drivers: lm t n
# Find hooked syscalls: !ssd

# Bootkit detection
# Check MBR / VBR / UEFI variables
bcdedit /enum firmware

Phase 6 — YARA rule authoring

rule Emotet_Loader_v4 {
    meta:
        author = "redteam"
        date = "2026-06-28"
        description = "Emotet v4 loader"
        reference = "https://attack.mitre.org/software/S0679/"
    strings:
        $s1 = "emotet" wide ascii nocase
        $s2 = { 6A 40 68 00 30 00 00 6A 14 8D 91 }
        $s3 = "%u%.4x" wide ascii
        $api1 = "CryptStringToBinaryA" wide
        $api2 = "InternetOpenA" wide
    condition:
        uint16(0) == 0x5A4D and
        3 of ($s*) and
        2 of ($api*)
}

Phase 7 — IDA Pro workflow

# IDA Python script: find anti-debug calls
import idautils, idc

for func_ea in idautils.Functions():
    name = idc.get_func_name(func_ea)
    if name in ["IsDebuggerPresent", "CheckRemoteDebuggerPresent"]:
        print(f"Anti-debug: {name} at {hex(func_ea)}")

# Decompile function
import ida_hexrays
cf = ida_hexrays.decompile(func_ea)
print(cf)

Phase 8 — Ghidra workflow

# Ghidra Python: find suspicious imports
from ghidra.program.model.symbol import SymbolType

sm = currentProgram.getSymbolTable()
for sym in sm.getAllSymbols(True):
    if sym.getSymbolType() == SymbolType.FUNCTION:
        name = sym.getName()
        if "VirtualProtect" in name or "WriteProcessMemory" in name:
            print(f"Inject: {name} at {sym.getAddress()}")

Phase 9 — Memory forensics (Volatility)

# Process listing
volatility -f memory.dmp windows.pslist

# Network connections
volatility -f memory.dmp windows.netscan

# Injected code detection
volatility -f memory.dmp windows.malfind

# DLL list per process
volatility -f memory.dmp windows.dlllist --pid 1234

# Kernel driver listing
volatility -f memory.dmp windows.modscan

Phase 10 — Reporting

Produce malware analysis report:

• Family + variant
• IOCs (hashes, domains, IPs, mutexes)
• TTP mapping (MITRE ATT&CK)
• YARA rules
• Detection recommendations

Practical Steps

Step 1 — Triage new sample

# Hash
sha256sum malware.exe > hash.txt

# VT lookup
curl -s "https://www.virustotal.com/api/v3/files/$(sha256sum malware.exe | cut -d' ' -f1)" \
  -H "x-apikey: $VT_KEY" | jq .

# PE analysis
pe-tree malware.exe

Step 2 — Unpack UPX sample

upx -d malware_packed.exe -o malware_unpacked.exe

# Verify
sha256sum malware_unpacked.exe
file malware_unpacked.exe
strings -a malware_unpacked.exe | grep -iE "http|dll"

Step 3 — Unpack VMProtect sample

# In x64dbg:
# 1. Load binary
# 2. Set memory breakpoint on .vmp section execution
# 3. Run until breakpoint
# 4. Step until OEP (look for typical MSVC entry point pattern)
# 5. Use Scylla plugin:
#    - Select process
#    - Click "IAT AutoSearch"
#    - Click "Get Imports"
#    - Click "Dump" → save unpacked.exe
#    - Click "Fix Dump" → fix IAT

Step 4 — Identify anti-VM

# Static
strings malware.exe | grep -iE "vmware|virtualbox|qemu"
strings malware.exe | grep -iE "vmware tools|vbox guest additions"

# Registry keys
strings malware.exe | grep -iE "SYSTEM\\\\CurrentControlSet\\\\Services\\\\VBoxGuest"

Step 5 — YARA rule authoring

rule BlackCat_ALPHV_Ransomware {
    meta:
        author = "redteam"
        description = "BlackCat/ALPHV Rust-based ransomware"
        reference = "https://attack.mitre.org/software/S1068/"
    strings:
        $rust = "rust_panic" wide ascii
        $s1 = "BlackCat" wide ascii nocase
        $s2 = "{ 52 75 73 74 }" // "Rust" in hex
        $api1 = "CryptEncrypt" wide
        $api2 = "BCryptEncrypt" wide
    condition:
        uint16(0) == 0x5A4D and
        $rust and
        any of ($s*) and
        any of ($api*)
}

Step 6 — Memory forensics

volatility -f memory.dmp windows.pslist | grep -v "Microsoft\|Windows"
volatility -f memory.dmp windows.netscan
volatility -f memory.dmp windows.malfind --pid 1234

Step 7 — Rootkit detection

# User-mode hook detection
pe-sieve /pid 1234 /imp 3

# Hollowed process detection
hollows-hunter /pid 1234

# Autoruns (autostart persistence)
autoruns -accepteula -a autostart.arn

Defense Perspective

Defenders must assume:

1. Packed malware evades AV signature — unpacking + behavioral detection required
2. Sandbox evasion defeats dynamic analysis — anti-VM must be bypassed
3. Rootkits hide in kernel — kernel-mode detection (PatchGuard, EDR) required
4. UEFI rootkits persist across reinstall — firmware scanning required
5. YARA rules need constant tuning — false positives / false negatives
6. Memory forensics catches fileless malware — Volatility essential
7. Threat actor tooling evolves rapidly — analyst workflow automation needed
8. Malware uses LOLBins — signed binaries (certutil, bitsadmin) bypass allowlist

Key defensive controls:

• Behavior-based detection (EDR / XDR)
• Memory scanning (pe-sieve, hollows-hunter)
• YARA scanning at egress + endpoint
• Volatility memory forensics for IR
• Application allowlisting (AppLocker, WDAC)
• Kernel-mode protection (PatchGuard)
• UEFI Secure Boot
• Behavioral baseline for processes

Packer Triage Cheat Sheet

Sandbox Evasion Techniques

Rootkit Categories

Threat Actor Tooling

Engagement Workflow

1. Triage — hash, file type, PE analysis, VT lookup
2. Static analysis — strings, section entropy, import analysis
3. Unpacking — UPX/manual/VMProtect devirt
4. Dynamic analysis — sandbox + manual x64dbg
5. Evasion triage — anti-VM, anti-debug, anti-analysis
6. Rootkit detection — user/kernel/boot/UEFI
7. YARA authoring — detection rules
8. Reporting — IOCs, TTPs, detection recommendations

Lab Setup

# Cuckoo sandbox
git clone https://github.com/cuckoosandbox/cuckoo
cd cuckoo && python3 setup.py install

# Ghidra
wget https://github.com/NationalSecurityAgency/ghidra/releases/download/Ghidra_11.0_build/ghidra_11.0_PUBLIC_20231222.zip
unzip ghidra_11.0_PUBLIC_20231222.zip

# Volatility
pip install volatility3

# YARA
pip install yara-python

Quality Checklist

• File hash + VT lookup
• PE analysis complete
• Packer identified
• Sample unpacked
• Sandbox evasion identified
• Rootkit techniques analyzed
• YARA rule authored
• IOCs documented
• TTP mapping complete
• Final report delivered

Detection Methods

Static Analysis Detection

• AV signatures: YARA rules, ClamAV signatures match known malware families.
• Entropy analysis: PE sections with entropy > 7.0 (packed/encrypted).
• Import table anomalies: Missing imports (LoadLibrary, GetProcAddress); imports via hash resolution.
• PE structure anomalies: Imports via only LoadLibraryA/GetProcAddress; signature of dynamic resolution.

Dynamic Analysis Detection

• Sandbox detonation: Cuckoo, Joe Sandbox; behavioral signatures.
• API call sequences: Mimikatz signature (OpenProcess + ReadProcessMemory + WriteProcessMemory on lsass).
• Network anomalies: Connections to known C2 infrastructure (Cobalt Strike teamserver default ports).

SIEM Detection Rules

• Splunk SPL: index=malware sourcetype=yara | where rule matches "mimikatz/*"
• YARA rules: Continuous scanning of filesystem for known signatures.
• AMSI integration: Scan PowerShell, VBA, JavaScript content via AMSI.

Defense Evasion Techniques

Anti-Analysis

• Anti-debugging: IsDebuggerPresent, ntdll!KdUserExceptionDispatcher check, timing checks (rdtsc).
• Anti-VM: MAC address check (VMware 00:50:56), CPUID hypervisor bit, registry artifacts.
• Anti-sandbox: Mouse movement check (real users have jitter), recent documents check, uptime check.
• Anti-AV: Process enumeration looking for av processes; exit if found.

Code Obfuscation

• Packing: UPX, ASPack, Themida, VMProtect; detect via entropy.
• Polymorphic code: Decryptor changes; payload signature constant.
• Metamorphic code: Body rewritten each generation; no static signature.
• Control flow flattening: Switch dispatcher; defeats static analysis.
• Junk code insertion: No-op instructions between real instructions.
• String encryption: Encrypt sensitive strings; decrypt at runtime only.

Memory-Resident Evasion

• Reflective DLL injection: Load DLL from memory; no file artifacts.
• Process hollowing: Replace legitimate process memory; appears as legitimate process.
• Module stomping: Load legitimate DLL, overwrite; inherits module legitimacy.
• Phantom DLL hollowing: Hollow rarely-used DLL; less attention.

Modern AV/EDR Bypass

• AMSI bypass: Patch amsi.dll!AmsiScanBuffer in-memory.
• ETW bypass: Patch ntdll!EtwEventWrite in-memory.
• Direct syscalls: Bypass user-mode hooks (SysWhispers, HellsGate).
• BYOVD: Load vulnerable signed driver for kernel R/W.

References

• MITRE ATT&CK Defense Evasion — https://attack.mitre.org/tactics/TA0005/
• "Practical Malware Analysis" (Sikorski, Honig, 2012)
• "Malware Analyst's Cookbook" (Ligh, Adair, Hartstein, 2010)
• "Learning Malware Analysis" (Monnappa Ka, 2018)
• "The Art of Memory Forensics" (Hale Ligh, 2014)
• IDA Pro Book (Chris Eagle, 2nd Edition)
• Ghidra documentation — https://ghidra-sre.org/
• Volatility documentation — https://volatility3.readthedocs.io/
• YARA documentation — https://yara.readthedocs.io/
• CISA AA21-148A — DarkSide/Conti analysis
• Mandiant APT1 / APT41 reports
• CrowdStrike 2024 Global Threat Report
• MalwareBazaar — https://bazaar.abuse.ch/
• VirusTotal — https://www.virustotal.com/
• Any.Run sandbox — https://app.any.run/
• JoeSandbox — https://www.joesandbox.com/
• "VMProtect Devirtualization" (BlackHat 2023)
• "Rootkit Arsenal" (Blunden, 3rd Edition)
• "UEFI Rootkits" (ESET LoJax report 2018)