# Malware Analysis Advanced

> Advanced malware analysis covering unpacking (UPX, VMProtect, Themida, Enigma, custom packers), sandbox-evasion detection (anti-VM, anti-debug, anti-analysis), rootkit analysis (user-mode, kernel-mode, bootkits, UEFI), YARA rule authoring and optimization, and IDA Pro / Ghidra / Binary Ninja workflows. Distinct from foundational `binary-reverse` — focuses on dynamic unpacking, evasion triage, rootkit techniques, and analyst workflow automation. Use when analyzing modern packed malware (Emotet, TrickBot, Conti, LockBit, BlackCat/ALPHV, REvil), authoring detection rules, or building automated malware triage pipelines.

- Skill: `brucesongs/malware-analysis-advanced` (Agent Skill, multi-file: 10 files)
- Install (CLI): `npx skillmds@latest add brucesongs/malware-analysis-advanced`
- Raw SKILL.md: https://api.skillmd.com/api/skills/brucesongs/malware-analysis-advanced/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: brucesongs (https://skillmd.com/u/brucesongs)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/brucesongs/malware-analysis-advanced

---


# Malware Analysis Advanced

## Summary

Advanced malware analysis is the discipline of unpacking, reverse engineering, and understanding modern packed/obfuscated malware (UPX, VMProtect, Themida, Enigma, custom packers), identifying sandbox-evasion techniques (anti-VM, anti-debug, anti-analysis), analyzing rootkits (user-mode, kernel-mode, bootkits, UEFI), and authoring detection rules (YARA). This domain covers full unpacking workflows (static + dynamic), modern threat actor tooling (Emotet, TrickBot, Conti, LockBit, BlackCat/ALPHV, REvil), YARA rule authoring and optimization, and industry-standard analyst tooling (IDA Pro, Ghidra, Binary Ninja, radare2). Distinct from foundational `binary-reverse` — this skill focuses on dynamic unpacking, evasion triage, rootkit techniques, and analyst workflow automation.

## Key Terms

- **Packer** — Tool that compresses + encrypts executable to evade AV (UPX, VMProtect, Themida)
- **Unpacking** — Recovering original executable from packed binary
- **Anti-VM** — Code that detects virtual machine (sandbox evasion)
- **Anti-debug** — Code that detects debugger presence (analyst evasion)
- **Anti-analysis** — Umbrella term for AV/VM/debug/sandbox evasion
- **Rootkit** — Tool that hides processes / files / network connections
- **User-mode rootkit** — Rootkit running in ring 3 (DLL injection, API hooking)
- **Kernel-mode rootkit** — Rootkit running in ring 0 (driver, syscall hooking)
- **Bootkit** — Rootkit that infects bootloader (MBR / VBR)
- **UEFI rootkit** — Rootkit that infects UEFI firmware (persistent across reinstall)
- **YARA** — Pattern-matching tool for malware identification
- **PE** — Portable Executable format (Windows binaries)
- **ELF** — Executable and Linkable Format (Linux binaries)
- **Triage** — Initial malware assessment (severity, family, capability)
- **Sandbox** — Isolated analysis environment (Cuckoo, JoeSandbox, Any.Run)

## Scope

This skill covers **advanced malware analysis**:
- Unpacking modern packers (UPX, VMProtect, Themida, Enigma, custom)
- Sandbox-evasion detection (anti-VM, anti-debug, anti-analysis)
- Rootkit analysis (user-mode, kernel-mode, bootkits, UEFI)
- YARA rule authoring and optimization
- IDA Pro / Ghidra / Binary Ninja workflows
- Modern threat tooling (Emotet, TrickBot, Conti, LockBit, BlackCat/ALPHV, REvil)
- Automated triage pipelines

**Out of scope**: foundational RE (see `binary-reverse`), exploit development (see `exploit-development`), reverse engineering theory (see `reverse-engineering-advanced`).

## Use Cases

- **Unpacking packed malware**: Recover original code from VMProtect/Themida binaries
- **Sandbox evasion triage**: Identify anti-VM/anti-debug techniques
- **Rootkit detection**: Find user-mode/kernel-mode rootkit techniques
- **YARA rule authoring**: Detect malware families by signature
- **Threat actor tooling analysis**: Reverse engineer Emotet, TrickBot, Conti, LockBit, BlackCat, REvil
- **Automated triage pipeline**: Build Cuckoo/JoeSandbox pipeline for mass triage
- **Detection rule optimization**: Tune YARA rules for performance
- **Memory forensics**: Use Volatility for in-memory malware analysis
- **UEFI analysis**: Identify UEFI rootkits (LoJax, MosaicRegressor)
- **API hook detection**: Find user-mode rootkit hooks

## Core Tools

| Tool | Purpose |
|------|---------|
| `IDA Pro` | Industry-standard disassembler + decompiler |
| `Ghidra` | Open-source RE tool (NSA) |
| `Binary Ninja` | Modern disassembler + decompiler |
| `radare2` | Open-source disassembler |
| `x64dbg` | Windows dynamic debugger |
| `WinDbg` | Windows kernel debugger |
| `gdb` | GNU Debugger (Linux) |
| `yara` | Pattern-matching tool for malware detection |
| `volatility3` | Memory forensics framework |
| `upx` | UPX unpacker |
| `vmprotect-devirt` | VMProtect devirtualization |
| `pe-sieve` | PE artifact scanner (process memory) |
| `hollows-hunter` | Process hollowing detector |
| `procdot` | Visual malware analysis |
| `process-hacker` | Process explorer (Windows) |
| `autoruns` | Autostart entry scanner |
| `pcap-ng-tools` | Network capture analysis |
| `zeek` | Network behavior analyzer |
| `suricata` | IDS / IPS for malware traffic |
| `pe-tree` | Visual PE analysis |
| `imhex` | Modern hex editor |

## Methodology

### Phase 1 — Static triage

```bash
# File hash
sha256sum malware.exe

# File type
file malware.exe

# PE analysis
pe-tree malware.exe

# Strings
strings -a malware.exe | grep -iE "http|dll|reg|cmd"

# Section entropy (packed indicator)
python3 -c "
import pefile
pe = pefile.PE('malware.exe')
for section in pe.sections:
    print(f'{section.Name.decode().strip():12s} entropy={section.get_entropy():.2f}')
"
```

### Phase 2 — Unpacking

```bash
# UPX
upx -d malware.exe -o malware_unpacked.exe

# VMProtect / Themida / custom
# Use x64dbg + Scylla to dump from memory
# 1. Load in x64dbg
# 2. Set breakpoint on OEP (original entry point)
# 3. Run until OEP hit
# 4. Use Scylla to dump process memory
# 5. Fix IAT (Import Address Table)
```

### Phase 3 — Dynamic analysis

```bash
# Run in sandbox (Cuckoo)
cuckoo submit malware.exe

# Manual analysis with x64dbg
# 1. Load binary
# 2. Set breakpoints on WinAPI calls (CreateFile, WriteFile, etc.)
# 3. Run, observe behavior
# 4. Capture network traffic (Wireshark)

# Volatility memory analysis
volatility -f memory.dmp windows.pslist
volatility -f memory.dmp windows.netscan
volatility -f memory.dmp windows.malfind
```

### Phase 4 — Sandbox evasion detection

```bash
# Static anti-VM strings
strings malware.exe | grep -iE "vmware|virtualbox|qemu|hyper-v|xen"
strings malware.exe | grep -iE "vbox|vmware tools|prl_"

# Anti-debug APIs
strings malware.exe | grep -iE "IsDebuggerPresent|CheckRemoteDebuggerPresent|NtQueryInformationProcess"

# Dynamic API tracing
# Use API monitor to capture all API calls
```

### Phase 5 — Rootkit analysis

```bash
# User-mode rootkit detection
pe-sieve /pid 1234 /imp 3
hollows-hunter /pid 1234

# Kernel-mode rootkit detection
# Use WinDbg kernel mode
# List loaded drivers: lm t n
# Find hooked syscalls: !ssd

# Bootkit detection
# Check MBR / VBR / UEFI variables
bcdedit /enum firmware
```

### Phase 6 — YARA rule authoring

```yara
rule Emotet_Loader_v4 {
    meta:
        author = "redteam"
        date = "2026-06-28"
        description = "Emotet v4 loader"
        reference = "https://attack.mitre.org/software/S0679/"
    strings:
        $s1 = "emotet" wide ascii nocase
        $s2 = { 6A 40 68 00 30 00 00 6A 14 8D 91 }
        $s3 = "%u%.4x" wide ascii
        $api1 = "CryptStringToBinaryA" wide
        $api2 = "InternetOpenA" wide
    condition:
        uint16(0) == 0x5A4D and
        3 of ($s*) and
        2 of ($api*)
}
```

### Phase 7 — IDA Pro workflow

```python
# IDA Python script: find anti-debug calls
import idautils, idc

for func_ea in idautils.Functions():
    name = idc.get_func_name(func_ea)
    if name in ["IsDebuggerPresent", "CheckRemoteDebuggerPresent"]:
        print(f"Anti-debug: {name} at {hex(func_ea)}")

# Decompile function
import ida_hexrays
cf = ida_hexrays.decompile(func_ea)
print(cf)
```

### Phase 8 — Ghidra workflow

```python
# Ghidra Python: find suspicious imports
from ghidra.program.model.symbol import SymbolType

sm = currentProgram.getSymbolTable()
for sym in sm.getAllSymbols(True):
    if sym.getSymbolType() == SymbolType.FUNCTION:
        name = sym.getName()
        if "VirtualProtect" in name or "WriteProcessMemory" in name:
            print(f"Inject: {name} at {sym.getAddress()}")
```

### Phase 9 — Memory forensics (Volatility)

```bash
# Process listing
volatility -f memory.dmp windows.pslist

# Network connections
volatility -f memory.dmp windows.netscan

# Injected code detection
volatility -f memory.dmp windows.malfind

# DLL list per process
volatility -f memory.dmp windows.dlllist --pid 1234

# Kernel driver listing
volatility -f memory.dmp windows.modscan
```

### Phase 10 — Reporting

Produce malware analysis report:
- Family + variant
- IOCs (hashes, domains, IPs, mutexes)
- TTP mapping (MITRE ATT&CK)
- YARA rules
- Detection recommendations

## Practical Steps

### Step 1 — Triage new sample

```bash
# Hash
sha256sum malware.exe > hash.txt

# VT lookup
curl -s "https://www.virustotal.com/api/v3/files/$(sha256sum malware.exe | cut -d' ' -f1)" \
  -H "x-apikey: $VT_KEY" | jq .

# PE analysis
pe-tree malware.exe
```

### Step 2 — Unpack UPX sample

```bash
upx -d malware_packed.exe -o malware_unpacked.exe

# Verify
sha256sum malware_unpacked.exe
file malware_unpacked.exe
strings -a malware_unpacked.exe | grep -iE "http|dll"
```

### Step 3 — Unpack VMProtect sample

```bash
# In x64dbg:
# 1. Load binary
# 2. Set memory breakpoint on .vmp section execution
# 3. Run until breakpoint
# 4. Step until OEP (look for typical MSVC entry point pattern)
# 5. Use Scylla plugin:
#    - Select process
#    - Click "IAT AutoSearch"
#    - Click "Get Imports"
#    - Click "Dump" → save unpacked.exe
#    - Click "Fix Dump" → fix IAT
```

### Step 4 — Identify anti-VM

```bash
# Static
strings malware.exe | grep -iE "vmware|virtualbox|qemu"
strings malware.exe | grep -iE "vmware tools|vbox guest additions"

# Registry keys
strings malware.exe | grep -iE "SYSTEM\\\\CurrentControlSet\\\\Services\\\\VBoxGuest"
```

### Step 5 — YARA rule authoring

```yara
rule BlackCat_ALPHV_Ransomware {
    meta:
        author = "redteam"
        description = "BlackCat/ALPHV Rust-based ransomware"
        reference = "https://attack.mitre.org/software/S1068/"
    strings:
        $rust = "rust_panic" wide ascii
        $s1 = "BlackCat" wide ascii nocase
        $s2 = "{ 52 75 73 74 }" // "Rust" in hex
        $api1 = "CryptEncrypt" wide
        $api2 = "BCryptEncrypt" wide
    condition:
        uint16(0) == 0x5A4D and
        $rust and
        any of ($s*) and
        any of ($api*)
}
```

### Step 6 — Memory forensics

```bash
volatility -f memory.dmp windows.pslist | grep -v "Microsoft\|Windows"
volatility -f memory.dmp windows.netscan
volatility -f memory.dmp windows.malfind --pid 1234
```

### Step 7 — Rootkit detection

```bash
# User-mode hook detection
pe-sieve /pid 1234 /imp 3

# Hollowed process detection
hollows-hunter /pid 1234

# Autoruns (autostart persistence)
autoruns -accepteula -a autostart.arn
```

### Defense Perspective

Defenders must assume:

1. **Packed malware evades AV signature** — unpacking + behavioral detection required
2. **Sandbox evasion defeats dynamic analysis** — anti-VM must be bypassed
3. **Rootkits hide in kernel** — kernel-mode detection (PatchGuard, EDR) required
4. **UEFI rootkits persist across reinstall** — firmware scanning required
5. **YARA rules need constant tuning** — false positives / false negatives
6. **Memory forensics catches fileless malware** — Volatility essential
7. **Threat actor tooling evolves rapidly** — analyst workflow automation needed
8. **Malware uses LOLBins** — signed binaries (certutil, bitsadmin) bypass allowlist

Key defensive controls:

- Behavior-based detection (EDR / XDR)
- Memory scanning (pe-sieve, hollows-hunter)
- YARA scanning at egress + endpoint
- Volatility memory forensics for IR
- Application allowlisting (AppLocker, WDAC)
- Kernel-mode protection (PatchGuard)
- UEFI Secure Boot
- Behavioral baseline for processes

## Packer Triage Cheat Sheet

| Packer | Detection | Unpacking difficulty |
|--------|-----------|---------------------|
| UPX | Section ".UPX0/.UPX1" | Easy (`upx -d`) |
| ASPack | Section ".aspack" | Medium (manual) |
| Themida | Section ".Themida" | Hard (WinDbg) |
| VMProtect | Section ".vmp0/.vmp1" | Very Hard (devirt) |
| Enigma | Section ".enigma1/.enigma2" | Hard |
| Custom | High entropy + obfuscation | Very Hard |

## Sandbox Evasion Techniques

| Technique | Detection | Bypass |
|-----------|-----------|--------|
| CPUID VM bit | Static strings | Patch CPUID |
| Registry VM keys | Strings (VMware, VBox) | Registry scrub |
| MAC address OUI | Network adapter | Spoof MAC |
| Process count | Psapi enumeration | Inject extra processes |
| Sleep + check | Timing analysis | Hook sleep |
| Mouse movement | Cursor position | Virtual mouse |
| Disk size | <60GB = VM | Larger VMDK |
| Recent files | User profile age | Pre-populate |

## Rootkit Categories

| Type | Ring | Persistence | Example |
|------|------|-------------|---------|
| User-mode | 3 | Registry | Hacker Defender |
| Kernel-mode | 0 | Driver | Rustock |
| Bootkit | 0 | MBR/VBR | TDL4 |
| UEFI | -1 | Firmware | LoJax |

## Threat Actor Tooling

| Family | Type | Packer | Notable Techniques |
|--------|------|--------|---------------------|
| Emotet | Loader | Custom | Macro dropper, polymorphic |
| TrickBot | Banking trojan | Custom | Process hollowing, anti-VM |
| Conti | Ransomware | Custom | LockBit-shared code, Rclone exfil |
| LockBit 3.0 | Ransomware | Custom | StealBit exfil, customizable |
| BlackCat/ALPHV | Ransomware | Rust | MEGA exfil, cross-platform |
| REvil | Ransomware | Custom | Affiliate program, onion leak |

## Engagement Workflow

1. **Triage** — hash, file type, PE analysis, VT lookup
2. **Static analysis** — strings, section entropy, import analysis
3. **Unpacking** — UPX/manual/VMProtect devirt
4. **Dynamic analysis** — sandbox + manual x64dbg
5. **Evasion triage** — anti-VM, anti-debug, anti-analysis
6. **Rootkit detection** — user/kernel/boot/UEFI
7. **YARA authoring** — detection rules
8. **Reporting** — IOCs, TTPs, detection recommendations

## Lab Setup

```bash
# Cuckoo sandbox
git clone https://github.com/cuckoosandbox/cuckoo
cd cuckoo && python3 setup.py install

# Ghidra
wget https://github.com/NationalSecurityAgency/ghidra/releases/download/Ghidra_11.0_build/ghidra_11.0_PUBLIC_20231222.zip
unzip ghidra_11.0_PUBLIC_20231222.zip

# Volatility
pip install volatility3

# YARA
pip install yara-python
```

## Quality Checklist

- [ ] File hash + VT lookup
- [ ] PE analysis complete
- [ ] Packer identified
- [ ] Sample unpacked
- [ ] Sandbox evasion identified
- [ ] Rootkit techniques analyzed
- [ ] YARA rule authored
- [ ] IOCs documented
- [ ] TTP mapping complete
- [ ] Final report delivered

## Detection Methods

### Static Analysis Detection
- **AV signatures**: YARA rules, ClamAV signatures match known malware families.
- **Entropy analysis**: PE sections with entropy > 7.0 (packed/encrypted).
- **Import table anomalies**: Missing imports (`LoadLibrary`, `GetProcAddress`); imports via hash resolution.
- **PE structure anomalies**: Imports via only `LoadLibraryA`/`GetProcAddress`; signature of dynamic resolution.

### Dynamic Analysis Detection
- **Sandbox detonation**: Cuckoo, Joe Sandbox; behavioral signatures.
- **API call sequences**: Mimikatz signature (OpenProcess + ReadProcessMemory + WriteProcessMemory on lsass).
- **Network anomalies**: Connections to known C2 infrastructure (Cobalt Strike teamserver default ports).

### SIEM Detection Rules
- **Splunk SPL**: `index=malware sourcetype=yara | where rule matches "mimikatz/*"`
- **YARA rules**: Continuous scanning of filesystem for known signatures.
- **AMSI integration**: Scan PowerShell, VBA, JavaScript content via AMSI.

## Defense Evasion Techniques

### Anti-Analysis
- **Anti-debugging**: `IsDebuggerPresent`, `ntdll!KdUserExceptionDispatcher` check, timing checks (rdtsc).
- **Anti-VM**: MAC address check (VMware 00:50:56), CPUID hypervisor bit, registry artifacts.
- **Anti-sandbox**: Mouse movement check (real users have jitter), recent documents check, uptime check.
- **Anti-AV**: Process enumeration looking for av processes; exit if found.

### Code Obfuscation
- **Packing**: UPX, ASPack, Themida, VMProtect; detect via entropy.
- **Polymorphic code**: Decryptor changes; payload signature constant.
- **Metamorphic code**: Body rewritten each generation; no static signature.
- **Control flow flattening**: Switch dispatcher; defeats static analysis.
- **Junk code insertion**: No-op instructions between real instructions.
- **String encryption**: Encrypt sensitive strings; decrypt at runtime only.

### Memory-Resident Evasion
- **Reflective DLL injection**: Load DLL from memory; no file artifacts.
- **Process hollowing**: Replace legitimate process memory; appears as legitimate process.
- **Module stomping**: Load legitimate DLL, overwrite; inherits module legitimacy.
- **Phantom DLL hollowing**: Hollow rarely-used DLL; less attention.

### Modern AV/EDR Bypass
- **AMSI bypass**: Patch `amsi.dll!AmsiScanBuffer` in-memory.
- **ETW bypass**: Patch `ntdll!EtwEventWrite` in-memory.
- **Direct syscalls**: Bypass user-mode hooks (SysWhispers, HellsGate).
- **BYOVD**: Load vulnerable signed driver for kernel R/W.

## References

- MITRE ATT&CK Defense Evasion — https://attack.mitre.org/tactics/TA0005/
- "Practical Malware Analysis" (Sikorski, Honig, 2012)
- "Malware Analyst's Cookbook" (Ligh, Adair, Hartstein, 2010)
- "Learning Malware Analysis" (Monnappa Ka, 2018)
- "The Art of Memory Forensics" (Hale Ligh, 2014)
- IDA Pro Book (Chris Eagle, 2nd Edition)
- Ghidra documentation — https://ghidra-sre.org/
- Volatility documentation — https://volatility3.readthedocs.io/
- YARA documentation — https://yara.readthedocs.io/
- CISA AA21-148A — DarkSide/Conti analysis
- Mandiant APT1 / APT41 reports
- CrowdStrike 2024 Global Threat Report
- MalwareBazaar — https://bazaar.abuse.ch/
- VirusTotal — https://www.virustotal.com/
- Any.Run sandbox — https://app.any.run/
- JoeSandbox — https://www.joesandbox.com/
- "VMProtect Devirtualization" (BlackHat 2023)
- "Rootkit Arsenal" (Blunden, 3rd Edition)
- "UEFI Rootkits" (ESET LoJax report 2018)