Hexcore Binary Analysis

HexCore Binary Analysis Skill — v3.7.1

Overview

HexCore is a VS Code fork for reverse engineering and binary analysis (HikariSystem HexCore). It includes 20+ extensions with 6 native engines (Capstone, Unicorn, Remill, LLVM MC, better-sqlite3, Helix) and a full automation pipeline with conditional branching.

Current version: v3.7.1 "Dynamic Intelligence + Pipeline Branching" (2026-03-14) Engine versions: capstone 1.3.2 | unicorn 1.2.1 | llvm-mc 1.0.0 | better-sqlite3 2.0.0 | remill 0.1.2 | helix 0.5.0 Deprecated: hexcore-rellic (superseded by Helix MLIR — removal planned for v3.8.0)

Extensions

Native Engines (no VS Code commands — pure API)

Disassembler (hexcore-disassembler v1.5.0)

Professional disassembler with Capstone engine, ELF/PE parsing, CFG, xrefs, patching, the pipeline runner, and advanced analysis (junk filtering, VM detection, PRNG detection).

Headless commands (pipeline-safe):

• hexcore.disasm.analyzeAll — Deep analysis (prolog scan + xrefs). New args: filterJunk, detectVM, detectPRNG
• hexcore.disasm.buildFormula — Symbolic expression extraction (x86/x64 only)
• hexcore.disasm.checkConstants — Validate numeric annotations
• hexcore.disasm.searchStringHeadless — Search string references
• hexcore.disasm.exportASMHeadless — Export assembly to file
• hexcore.disasm.disassembleAtHeadless — Disassemble N instructions at address. New args: filterJunk
• hexcore.disasm.liftToIR — Lift to LLVM IR (Remill, x86/x64/ARM64)
• hexcore.disasm.dumpAndDisassemble — Dump emulation memory + disassemble in one step (v3.7.1)
• hexcore.pipeline.runJob — Run automation job (now with onResult conditional branching)
• hexcore.pipeline.listCapabilities — Export capability map
• hexcore.pipeline.validateJob — Preflight validation
• hexcore.pipeline.validateWorkspace — Batch validation
• hexcore.pipeline.createPresetJob — Generate job from preset
• hexcore.pipeline.saveJobAsProfile — Save job as profile
• hexcore.pipeline.doctor — Diagnose health

Analysis features (v3.7.1):

• filterJunkInstructions() — Detect and remove 7 junk patterns (callfuscation, nop sleds, identity ops)
• detectVM() — VM obfuscation heuristics (dispatcher, handler tables, operand stacks)
• detectPRNG() — Static PRNG pattern detection (srand/rand call sites, seed extraction)
• loadBuffer() — Accept raw buffer for disassembly without file on disk (runtime memory)

Interactive commands (need UI):

• hexcore.disasm.openFile, analyzeFile, goToAddress, findXrefs, addComment, renameFunction, showCFG, searchString, exportASM, patchInstruction, nopInstruction, assemble, assembleMultiple, savePatchedFile, setSyntax, showLlvmVersion, nativeStatus

Experimental:

• hexcore.disasm.liftToIR — Lift to LLVM IR (requires Remill, x86/x64/ARM64 only)

Architecture auto-detection: Reads ELF e_machine / PE Machine headers. Supports x86, x64, ARM, ARM64, MIPS. Defaults to x64 for raw files.

Debugger (hexcore-debugger v2.2.0)

Emulation-based debugger using Unicorn engine with PE/ELF loading, API hooking, syscall handling, API call tracing, faithful PRNG emulation, side-channel analysis, and breakpoint auto-snapshots.

Process isolation & Smart Sync: x64 ELF and ARM64 ELF emulation run in dedicated child processes (x64ElfWorker.js, arm64Worker.js) to prevent Unicorn heap corruption from crashing the VS Code extension host. The worker communicates via JSON-RPC over IPC. A unique Smart Sync architecture instantly synchronizes heap memory (e.g. dynamically allocated strings) from the Worker to the Host before evaluating any API hook (such as __printf_chk, getline, or puts), guaranteeing flawless validation of complex obfuscated VMs (like active advanced HTB CTFs). PE emulation and other architectures run in-process.

Headless commands (pipeline-safe):

• hexcore.debug.emulateFullHeadless — Unified single-shot emulation (load → configure → run → collect → dispose). New v3.7.1 args: permissiveMemoryMapping, prngMode, prngSeed, collectSideChannels, memoryDumps, breakpointConfigs (with autoSnapshot). Aliases: hexcore.debug.emulate.full, hexcore.debug.run
• hexcore.debug.writeMemoryHeadless — Write data to emulation memory
• hexcore.debug.setRegisterHeadless — Set CPU register value
• hexcore.debug.setStdinHeadless — Set STDIN buffer for emulation
• hexcore.debug.disposeHeadless — Dispose emulation session (idempotent)
• hexcore.debug.snapshotHeadless — Save emulation snapshot
• hexcore.debug.restoreSnapshotHeadless — Restore emulation snapshot
• hexcore.debug.exportTraceHeadless — Export API/libc call trace as JSON

v3.7.1 Emulation Features:

• Permissive Memory Mapping — permissiveMemoryMapping: true maps all segments with RWX permissions, allowing self-modifying VMs to jump to .rodata/.data without UC_ERR_FETCH_PROT
• PRNG Modes — prngMode: 'glibc' (344-state TYPE_3 algorithm), 'msvcrt' (LCG: seed * 214013 + 2531011), 'stub' (returns 0, default). Faithful implementations that match native rand() sequences for any seed.
• Memory Dumps — memoryDumps: [{ address, size, trigger: 'breakpoint'|'end' }] captures arbitrary memory ranges during emulation
• Breakpoint Auto-Snapshots — breakpointConfigs: [{ address, autoSnapshot: true, dumpRanges? }] automatically captures registers, stack, and optional memory ranges at breakpoints, then continues execution
• Side-Channel Analysis — collectSideChannels: true installs instrumentation hooks to collect instruction counts per basic block, memory access patterns, and branch statistics
• Runtime Memory Disassembly — dumpAndDisassemble(address, size) combines memory reading and Capstone disassembly in one operation for analyzing runtime-decrypted code

Interactive commands (need UI):

• hexcore.debug.emulate — Start emulation (auto-detect arch)
• hexcore.debug.emulateWithArch — Start with manual arch selection
• hexcore.debug.emulationStep — Step one instruction
• hexcore.debug.emulationContinue — Continue to breakpoint/end
• hexcore.debug.emulationBreakpoint — Set breakpoint
• hexcore.debug.emulationReadMemory — Read memory region
• hexcore.debug.setStdin — Set STDIN buffer for ELF emulation
• hexcore.debug.saveSnapshot — Save emulation snapshot
• hexcore.debug.restoreSnapshot — Restore snapshot
• hexcore.debug.unicornStatus — Show Unicorn status

Internal engine capabilities (programmatic, not exposed as headless commands):

• PE loading with import resolution and Windows API hooks
• ELF loading with PLT stubs and Linux API hooks (libc emulation)
• Linux syscall handler (x86/x64: int 0x80, syscall instruction; ARM64: SVC #0)
• Architecture auto-detection from ELF/PE headers
• Deterministic ELF continue (250K instruction budget)
• STDIN buffer injection for scanf/read emulation
• Snapshot save/restore via Unicorn context
• x64 ELF worker process isolation with Smart Sync (prevents host heap corruption & guarantees dynamic string visibility)
• ARM64 ELF worker process isolation (same pattern)

Architecture support in debugger:

Other Extensions

Pipeline Automation

Creating Jobs

1. From preset: Run hexcore.pipeline.createPresetJob — choose quick-triage, full-static, or ctf-reverse
2. Manual: Create .hexcore_job.json in workspace root (see docs/HEXCORE_JOB_TEMPLATES.md)
3. Save profile: Run hexcore.pipeline.saveJobAsProfile to store in .hexcore_profiles.json

Running Jobs

• Auto: HexCore watches .hexcore_job.json and runs on create/change
• Manual: Run hexcore.pipeline.runJob
• Validate first: Run hexcore.pipeline.validateJob for preflight check
• Conditional branching: Use onResult in pipeline steps to skip, goto, abort, or log based on step output (v3.7.1)

Job Contract

Every headless command receives:

• file — path to target binary
• quiet — suppress UI notifications
• output — { path, format } for writing results

Output

Jobs produce in outDir:

• hexcore-pipeline.log — execution log with timestamps
• hexcore-pipeline.status.json — structured status per step (ok/failed/timed-out)
• Per-step output files (JSON or MD)

Architecture Support Matrix

Known Gaps (Critical for Agents)

1. Debugger interactive commands still need UI — MOSTLY RESOLVED: emulateFullHeadless provides full headless emulation (load → run → collect → dispose) without UI. writeMemoryHeadless, setRegisterHeadless, setStdinHeadless, and disposeHeadless fill remaining gaps. Only emulateWithArch (manual arch picker) remains interactive.
2. Debugger ARM64 ELF is incomplete — RESOLVED in v3.5.1: Full ARM64 DebugEngine with stack initialization, process stack layout (argc/argv via X0/X1/X2), SVC syscall handler, register state mapping, and 20+ Linux syscalls.
3. Debugger + static ELF — statically-linked binaries have no PLT stubs, so LinuxApiHooks cannot intercept libc calls. Only direct syscall interception works (and only for x86/x64/ARM64).
4. buildFormula is x86/x64 only — the register regex doesn't recognize ARM64 registers (x0-x30, sp, lr). (ARM64 formulaBuilder added in v3.5.1 but limited to 15 mnemonics)
5. No ELF analyzer extension — RESOLVED in v3.5.2: hexcore-elfanalyzer provides full ELF analysis (sections, segments, symbols, security mitigations).
6. Base64 decode has no headless mode — RESOLVED in v3.5.2: hexcore.base64.decodeHeadless is pipeline-safe.
7. Hex viewer has no headless dump — RESOLVED in v3.5.2: hexcore.hexview.dumpHeadless and hexcore.hexview.searchHeadless are pipeline-safe.
8. Strings XOR is 1-byte only — RESOLVED in v3.5.2: extractAdvanced now supports multi-byte XOR (2, 4, 8, 16 bytes), rolling XOR, and XOR with increment.
9. Prebuilds are win32-x64 only — Linux/macOS need node-gyp rebuild fallback.
10. Rellic is DEPRECATED — Superseded by Helix MLIR engine in v3.7.0. Remains functional for backward compatibility but will be removed in v3.8.0. Use hexcore.helix.decompile / hexcore.helix.decompileIR instead.

What Agents CAN Do

1. Create .hexcore_job.json files and run analysis via hexcore.pipeline.runJob
2. Use onResult conditional branching in pipeline steps to build adaptive workflows (skip, goto, abort, log)
3. Read pipeline output from hexcore-pipeline.status.json and step output files
4. Interpret results — entropy reports, string extractions, YARA matches, IOC lists
5. Validate jobs with hexcore.pipeline.validateJob before execution
6. Use presets via hexcore.pipeline.createPresetJob for quick setup
7. Search strings headlessly via hexcore.disasm.searchStringHeadless
8. Export assembly headlessly via hexcore.disasm.exportASMHeadless
9. Analyze ELF binaries via hexcore.elfanalyzer.analyze (sections, segments, symbols, security)
10. Decode Base64 via hexcore.base64.decodeHeadless
11. Dump hex ranges via hexcore.hexview.dumpHeadless
12. Search hex patterns via hexcore.hexview.searchHeadless
13. Run full emulation headlessly via hexcore.debug.emulateFullHeadless with permissive memory mapping, PRNG modes, side-channel analysis, memory dumps, and breakpoint auto-snapshots
14. Write emulation memory via hexcore.debug.writeMemoryHeadless
15. Set CPU registers via hexcore.debug.setRegisterHeadless
16. Set STDIN buffer via hexcore.debug.setStdinHeadless
17. Dispose emulation sessions via hexcore.debug.disposeHeadless
18. Save/restore emulation snapshots via hexcore.debug.snapshotHeadless / restoreSnapshotHeadless
19. Export API call traces via hexcore.debug.exportTraceHeadless
20. Compose unified reports via hexcore.pipeline.composeReport
21. Filter junk instructions via filterJunk: true in analyzeAll / disassembleAtHeadless args
22. Detect VM obfuscation via detectVM: true in analyzeAll args
23. Detect PRNG patterns via detectPRNG: true in analyzeAll args
24. Dump and disassemble runtime memory via dumpAndDisassemble for analyzing decrypted code
25. Decompile to pseudo-C via hexcore.helix.decompile (one-step) or liftToIR + hexcore.helix.decompileIR (two-step)

What Agents CANNOT Do

1. Start emulation — RESOLVED: Use hexcore.debug.emulateFullHeadless (or aliases hexcore.debug.emulate.full / hexcore.debug.run) for headless emulation. Interactive emulateWithArch still requires UI for manual arch selection.
2. See webviews — CFG graph, hex viewer, debugger view are visual only
3. Use interactive commands — file pickers, input boxes, quick-picks
4. Patch binaries — patchInstruction, nopInstruction, savePatchedFile need the disassembler UI open
5. Run YARA quick scan — requires prior UI context

Workflow: Static Analysis

1. hexcore.filetype.detect         → Identify file type
2. hexcore.hashcalc.calculate      → Compute hashes (VT lookup)
3. hexcore.entropy.analyze         → Detect packing/encryption
4. hexcore.strings.extract         → Extract strings
5. hexcore.strings.extractAdvanced → XOR deobfuscation (1-byte + multi-byte + rolling + increment) + stack strings
6. hexcore.base64.decodeHeadless   → Detect Base64 encoded strings
7. hexcore.hexview.dumpHeadless    → Inspect file header bytes
8. hexcore.peanalyzer.analyze      → PE headers/imports (PE files only)
9. hexcore.elfanalyzer.analyze     → ELF sections/segments/symbols/security (ELF files only)
10. hexcore.disasm.analyzeAll      → Deep disassembly + xrefs
11. hexcore.yara.scan              → Threat detection
12. hexcore.ioc.extract            → IOC extraction
13. hexcore.pipeline.composeReport → Unified report

Workflow: CTF Reverse Engineering

1. hexcore.filetype.detect              → Verify binary format
2. hexcore.disasm.analyzeAll            → Function discovery + xrefs
3. hexcore.disasm.exportASMHeadless     → Full disassembly export
4. hexcore.disasm.searchStringHeadless  → Find flag patterns
5. hexcore.strings.extractAdvanced      → Find obfuscated strings (multi-byte XOR, rolling, increment)
6. hexcore.base64.decodeHeadless        → Find Base64 encoded data
7. hexcore.hexview.searchHeadless       → Search for flag byte patterns
8. hexcore.disasm.buildFormula          → Extract key computations (x86/x64 only)

Workflow: Dynamic Analysis (Emulation)

1. hexcore.debug.emulateFullHeadless    → Single-shot emulation (recommended for pipeline jobs)
   Args: { file, arch?, stdin?, maxInstructions?, breakpoints?, keepAlive?,
           permissiveMemoryMapping?, prngMode?, prngSeed?,
           collectSideChannels?, memoryDumps?, breakpointConfigs?, output? }
   Returns: FullEmulationResult with registers, apiCalls, stdout, memoryRegions,
            crash status, snapshots, dumps, sideChannels

For advanced multi-step emulation (keepAlive: true):
2. hexcore.debug.emulateFullHeadless    → Start with keepAlive: true
3. hexcore.debug.writeMemoryHeadless    → Patch memory (base64 or 0x hex data)
4. hexcore.debug.setRegisterHeadless    → Modify CPU registers
5. hexcore.debug.setStdinHeadless       → Inject STDIN input
6. hexcore.debug.snapshotHeadless       → Save state checkpoint
7. hexcore.debug.disposeHeadless        → Clean up session

File Format Support

HexCore v3.7.1 "Dynamic Intelligence + Pipeline Branching" — Powered by Capstone 1.3.2 / Unicorn 1.2.1 / LLVM MC 1.0.0 / Remill 0.1.2 / Helix 0.5.0

Converted and distributed by TomeVault — claim your Tome and manage your conversions.