# Peak Threat Hunting V2

> Conduct threat hunts in Splunk using the PEAK framework (Prepare, Execute, Act with Knowledge) and create Notable Events for findings. Supports hypothesis-driven, baseline, and model-assisted hunts. Automatically generates Notable Events in Splunk Enterprise Security via sendalert. Use when the user wants to threat hunt, investigate security anomalies, baseline data sources, detect adversary techniques, map findings to MITRE ATT&CK, create notable events, or generate correlation search results.

- Skill: `mayeack/peak-threat-hunting-v2-4` (Agent Skill, multi-file: 4 files)
- Install (CLI): `npx skillmds@latest add mayeack/peak-threat-hunting-v2-4`
- Raw SKILL.md: https://api.skillmd.com/api/skills/mayeack/peak-threat-hunting-v2-4/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: mayeack (https://skillmd.com/u/mayeack)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/mayeack/peak-threat-hunting-v2-4

---


# PEAK Threat Hunting in Splunk (V2 — Notable Events)

Guide threat hunts using the PEAK framework, executing SPL queries via the Splunk MCP server. This version adds automatic Notable Event creation in Splunk ES for hunt findings.

## Quick Start

When a user requests a threat hunt:

1. **Identify hunt type** based on their goal:
   - **Hypothesis-driven**: Hunting specific adversary behavior (e.g., "Hunt for DNS tunneling")
   - **Baseline**: Understanding normal to find anomalies (e.g., "Baseline our firewall logs")
   - **Model-Assisted (M-ATH)**: Using ML/statistical models (e.g., "Find anomalous login patterns")

2. **Follow the PEAK phases**: Prepare → Execute → Act

3. **Run SPL queries** using the Splunk MCP server's `run_query` tool (a logical name — see [Splunk MCP Server Usage](#splunk-mcp-server-usage) to resolve it to your server's actual tool)

4. **Track progress** using the hunt checklist for the selected type

5. **Create Notable Events** for confirmed findings (Act phase — requires ES)

## Hunt Type Selection

```
Is this topic-based or data-based?
├── Topic-based (specific behavior/threat) → Does it have implicit complexity?
│   ├── Yes (many variables, ML needed) → Model-Assisted Hunt
│   └── No (explicit detection logic) → Hypothesis-Driven Hunt
└── Data-based (understand a data source) → Baseline Hunt
```

---

## ES Availability Check

Before creating Notable Events, verify that Splunk Enterprise Security is installed.

```
Action: Run via Splunk MCP
Tool: run_query
Query: | rest /services/apps/local/SplunkEnterpriseSecuritySuite | table title, version
Purpose: Confirm ES is installed
```

- **ES found**: Proceed with Notable Event creation in the Act phase.
- **ES not found**: Warn the user that Notable Events cannot be created. Continue with standard findings presentation, dashboards, and documentation only.

Cache this result for the duration of the hunt session — do not re-check for every finding.

---

## Hypothesis-Driven Hunt

Use when hunting for specific adversary behaviors or techniques.

### Phase 1: Prepare

```
Hunt Preparation Checklist:
- [ ] Topic selected and scoped
- [ ] Research completed (TTPs, threat intel)
- [ ] Hypothesis generated (testable statement)
- [ ] ABLE framework applied
- [ ] Data sources identified
- [ ] Hunt plan created
```

**1. Select Topic**: Identify the behavior to hunt (e.g., "lateral movement", "credential theft")

**2. Research Topic**: Gather threat intelligence:
- Known techniques for implementing this tactic
- Existing detections and gaps
- Threat actor TTPs (check MITRE ATT&CK)
- Prior hunts on this topic

**3. Generate Hypothesis**: Create a testable, falsifiable statement:
```
Template: "[Actor] may be [behavior] in [location] using [technique]"
Example: "An adversary may be exfiltrating data via DNS tunneling from finance endpoints"
```

**4. Apply ABLE Framework** to scope the hunt:

| Component | Question | Example |
|-----------|----------|---------|
| **A**ctor | Who/what threat? | APT group, insider, malware family |
| **B**ehavior | What TTP? | DNS tunneling, credential dumping |
| **L**ocation | Where to look? | Finance endpoints, DMZ servers |
| **E**vidence | What data/indicators? | DNS logs, unusually long queries |

**5. Check Data Availability**: Query available indexes and data:

```
Action: Run via Splunk MCP
Tool: list_indexes
Purpose: Identify available data sources
```

```
Action: Run via Splunk MCP
Tool: run_query
Query: | tstats count WHERE index=* by index, sourcetype | sort -count
Purpose: See data volume by sourcetype
```

### Phase 2: Execute

```
Hunt Execution Checklist:
- [ ] Data gathered and filtered
- [ ] Data pre-processed if needed
- [ ] Analysis techniques applied
- [ ] Hypothesis refined (if needed)
- [ ] Critical findings escalated
```

**1. Gather Data**: Collect evidence based on ABLE scope

**2. Pre-Process Data**: Ensure data quality:
- Normalize timestamps to UTC
- Filter to relevant scope
- Handle missing values

**3. Analyze**: Apply hunting techniques:

| Technique | When to Use | SPL Pattern |
|-----------|-------------|-------------|
| Stack counting (LFO) | Find rare values | `| stats count by field | sort count` |
| Frequency analysis | Find unusual patterns | `| timechart count by field` |
| Clustering | Group similar events | `| cluster field=field_name` |
| Outlier detection | Find statistical anomalies | `| eventstats avg(x) stdev(x) | eval z=(x-avg)/stdev` |

**4. Refine Hypothesis**: If initial analysis is inconclusive, adjust scope or indicators

**5. Escalate Critical Findings**: Immediately escalate confirmed malicious activity

### Phase 3: Act

```
Hunt Completion Checklist:
- [ ] Hunt preserved (queries, data, findings)
- [ ] Findings documented
- [ ] Notable Events created for confirmed findings (if ES available)
- [ ] Detections created/updated
- [ ] Dashboard created (if applicable)
- [ ] New hunt ideas logged
- [ ] Findings communicated to stakeholders
```

**1. Preserve Hunt**: Save queries, screenshots, and methodology

**2. Document Findings**: Use template in [HUNT_TEMPLATES.md](HUNT_TEMPLATES.md)

**3. Create Notable Events**: Follow the [Notable Event Creation Workflow](#notable-event-creation-workflow) below for each finding. Ask the user per-finding whether to create a Notable Event.

**4. Create Detections**: Convert findings to automated detection (see Detection Hierarchy below)

**5. Create Dashboard**: If the hunt produced findings that warrant ongoing monitoring or analyst review (Detection Hierarchy Level 2), invoke the `splunk-dashboard-studio` skill. Pass it:
- All SPL queries used during the Execute phase (both successful and refined versions)
- The hunt hypothesis and ABLE scope (Actor, Behavior, Location, Evidence)
- Key findings: field names, notable values, thresholds discovered
- The time range used during the hunt
- Any severity classifications or risk scores identified
- Context about the target audience (SOC analysts, hunt team, management)

**6. Communicate**: Share with SOC, IR, detection engineering teams

---

## Baseline Hunt

Use when establishing normal behavior for a data source to identify anomalies.

### Phase 1: Prepare

```
Baseline Preparation Checklist:
- [ ] Data source selected
- [ ] Documentation/schema researched
- [ ] Scope defined (systems, timeframe)
- [ ] Analysis plan created
```

**1. Select Data Source**: Prioritize by security relevance or coverage gaps

**2. Research Data Source**: Understand field meanings and expected values:

```
Action: Run via Splunk MCP
Tool: run_query
Query: | metadata type=sourcetypes index=<target_index> | table sourcetype, totalCount, lastTime
Purpose: Understand sourcetype coverage
```

**3. Scope**: Define system groups and timeframe (typically 30-90 days)

### Phase 2: Execute

```
Baseline Execution Checklist:
- [ ] Data gathered and filtered
- [ ] Data dictionary created
- [ ] Distributions reviewed
- [ ] Outliers investigated
- [ ] Gaps identified
- [ ] Relationships mapped
```

**1. Create Data Dictionary**: Document key fields:

```
Action: Run via Splunk MCP
Tool: run_query
Query: index=<target> | head 1000 | fieldsummary | table field, count, distinct_count, is_exact, numeric_count
Purpose: Understand field characteristics
```

**2. Review Distributions**: Establish baselines:

```spl
index=<target> | stats count by <field> | sort -count | head 20

index=<target> | stats avg(<field>) median(<field>) stdev(<field>) min(<field>) max(<field>)

index=<target> | stats dc(<field>) as unique_values
```

**3. Investigate Outliers**: Apply detection techniques from SPL_PATTERNS.md

**4. Gap Analysis**: Document missing data or quality issues

**5. Identify Relationships**: Map field correlations

### Phase 3: Act

Document baseline including:
- Data dictionary
- Statistical summaries
- Known benign outliers
- Detection candidates

**Create Notable Events**: If the baseline hunt identified security-relevant outliers, follow the [Notable Event Creation Workflow](#notable-event-creation-workflow) for each finding. Ask the user per-finding whether to create a Notable Event.

**Create Baseline Dashboard**: Baseline hunts are especially well suited for dashboards. Invoke the `splunk-dashboard-studio` skill to create a baseline monitoring dashboard. Pass it:
- The distribution queries (categorical, numeric, cardinality)
- The data dictionary fields and their characteristics
- Outlier thresholds discovered during the hunt
- The target index, sourcetypes, and scope
- Time range for the baseline period
The resulting dashboard gives analysts a reference view of "normal" to compare against during future investigations.

---

## Model-Assisted Hunt (M-ATH)

Use when hunting requires ML/statistical models to find complex patterns.

### Phase 1: Prepare

```
M-ATH Preparation Checklist:
- [ ] Topic selected with ML approach identified
- [ ] Research completed (methods, datasets)
- [ ] Datasets identified (labeled if supervised)
- [ ] Algorithm(s) selected
```

**Algorithm Selection Guide**:

| Goal | Algorithm Family | Splunk Approach |
|------|------------------|-----------------|
| Classify known threats | Classification | MLTK classifiers |
| Find unusual events | Anomaly detection | MLTK + `| anomalydetection` |
| Group similar events | Clustering | `| cluster` or MLTK |
| Predict future values | Time series | `| predict` |

### Phase 2: Execute

```
M-ATH Execution Checklist:
- [ ] Data gathered and preprocessed
- [ ] Model developed and refined
- [ ] Model applied to hunt data
- [ ] Results analyzed
- [ ] Critical findings escalated
```

**Key SPL for M-ATH**:

```spl
index=<target> | anomalydetection <field> action=annotate

index=<target> | timechart count | predict count future_timespan=24

index=<target> | cluster field=<text_field> showcount=true
```

### Phase 3: Act

Same as hypothesis-driven, but also:
- Preserve trained models
- Document model parameters and accuracy
- Consider model-based alerting
- Invoke the `splunk-dashboard-studio` skill to create a model monitoring dashboard that tracks model outputs, anomaly scores, and prediction accuracy over time

**Create Notable Events**: Follow the [Notable Event Creation Workflow](#notable-event-creation-workflow) for model-identified findings. Ask the user per-finding whether to create a Notable Event.

---

## Notable Event Creation Workflow

This workflow creates immediate Notable Events in Splunk ES for hunt findings. It runs during the Act phase after findings are presented to the user.

**Prerequisite**: ES must be available (see [ES Availability Check](#es-availability-check)). If ES is not installed, skip this workflow entirely.

### Step 1: Propose Notable Event Fields

For each finding, auto-derive the following fields and present them to the user for review:

| Field | Derivation |
|-------|------------|
| `rule_name` | `"PEAK Hunt: "` + hunt topic + `" - "` + finding title |
| `rule_description` | Finding description text |
| `security_domain` | Inferred from data source: `network` (firewall/proxy/DNS), `endpoint` (process/sysmon), `access` (auth logs), `identity` (user/account), `audit` (config/change) |
| `severity` | From the finding severity: `critical`, `high`, `medium`, `low`, `informational` |
| `drilldown_search` | The SPL query that produced the finding |
| `src` | Source IP/host from finding evidence (if available) |
| `dest` | Destination IP/host from finding evidence (if available) |
| `user` | Username from finding evidence (if available) |
| `mitre_attack_id` | MITRE ATT&CK technique ID if the hunt was ATT&CK-tagged |

### Step 2: User Confirms or Adjusts

Present the proposed fields to the user and ask:
1. Whether to create a Notable Event for this finding
2. Whether any field values should be adjusted

Use the **AskUserQuestion** tool for this confirmation so the user can accept or adjust in one step; fall back to asking conversationally if it is unavailable.

Creating a Notable Event writes to Splunk ES and pages on-call downstream. Never create one without explicit user confirmation for that specific finding. If the user declines, skip to the next finding.

### Step 3: Execute sendalert notable

Build and run the SPL query via the Splunk MCP:

```
Action: Run via Splunk MCP
Tool: run_query
Query:
| makeresults
| eval rule_name="<rule_name>",
       rule_description="<rule_description>",
       security_domain="<security_domain>",
       severity="<severity>",
       drilldown_search="<drilldown_search>",
       src="<src>",
       dest="<dest>",
       user="<user>",
       mitre_attack_id="<mitre_attack_id>"
| sendalert notable
Purpose: Create Notable Event for finding
```

- On **success**: Confirm to the user that the Notable Event was created, including the `rule_name` and `severity`.
- On **error**: Report the error to the user. Common issues: ES not installed, insufficient permissions, or the `notable` alert action not configured.

### Step 4: Record in Hunt Documentation

Add the created Notable Event to the hunt report (see [HUNT_TEMPLATES.md](HUNT_TEMPLATES.md) Notable Events section).

---

## Detection Hierarchy

Convert hunt findings to automated detection. Choose the highest feasible level:

| Level | Type | When to Use | Example |
|-------|------|-------------|---------|
| 4 (Best) | **Signatures/Rules** | High confidence, low false positives | ES correlation search |
| 3 | **Analytics in Code** | Complex logic, needs computation | MLTK model, Python script |
| 2 | **Dashboards** | Summarized for analyst review | Daily anomaly dashboard |
| 1 (Lowest) | **Reports** | High false positives, needs expertise | Weekly outlier report |

**For Level 4 (Signatures/Rules)**: During this hunt, immediate Notable Events are created via `| sendalert notable` for confirmed findings. For persistent automated detection, recommend the user create a scheduled Correlation Search in ES using the hunt's SPL query.

**For Level 2 (Dashboards)**: Always invoke the `splunk-dashboard-studio` skill. It will take your hunt queries and findings and produce a validated dashboard with appropriate visualizations, drilldown, and filtering.

---

## MITRE ATT&CK Integration

Map hunts to ATT&CK for coverage tracking. See [MITRE_MAPPING.md](MITRE_MAPPING.md) for technique-specific hunting guidance.

**Query ATT&CK-tagged events** (if ES is deployed):

```spl
index=notable | stats count by mitre_attack_technique_id | sort -count
```

---

## Hunt Metrics

Track these for each hunt:

| Metric | What to Record |
|--------|----------------|
| Detections created/updated | Count and IDs |
| Notable Events created | Count, rule_names, severities |
| Incidents opened | During hunt + from new detections |
| Gaps identified/closed | Data, visibility, tooling gaps |
| Vulnerabilities found | Misconfigs, missing patches |
| Techniques hunted | ATT&CK IDs covered |

See [HUNT_TEMPLATES.md](HUNT_TEMPLATES.md) for metrics tracking template.

---

## Splunk MCP Server Usage

Always use the Splunk MCP server to run queries. This skill refers to tools by **logical name** — different Splunk MCP servers expose them under different names, so at the start of a hunt, inspect the connected server's tool list and resolve each logical name to the actual tool. Do not guess: if no tool matches a capability, tell the user which one is missing.

| Logical name | Capability | Common actual names |
|--------------|------------|---------------------|
| `run_query` | Execute an SPL search | `run_splunk_query`, `splunk_run_query`, `search` |
| `list_indexes` | List available indexes | `get_indexes`, `splunk_get_indexes` |
| `list_knowledge_objects` | List saved searches, alerts, macros | `get_knowledge_objects`, `splunk_get_knowledge_objects` |

Claude Code namespaces MCP tools as `mcp__<server>__<tool>`. If the tools are deferred, load them with ToolSearch first.

```
Tool: run_query
Parameters:
  - query: SPL query string (required)
  - earliest_time: Start time (default: -24h)
  - latest_time: End time (default: now)
  - row_limit: Max results (default: 100, max: 1000)
```

**Other useful tools**:
- `list_indexes`: List available indexes
- `list_knowledge_objects`: List saved searches, alerts, macros, etc.

**Notable Event creation pattern** (requires ES):
```
Tool: run_query
Query: | makeresults | eval rule_name="...", severity="...", ... | sendalert notable
Purpose: Create an immediate Notable Event for a hunt finding
```

---

## Additional Resources

- [HUNT_TEMPLATES.md](HUNT_TEMPLATES.md) - Documentation templates
- [SPL_PATTERNS.md](SPL_PATTERNS.md) - Common hunting SPL patterns
- [MITRE_MAPPING.md](MITRE_MAPPING.md) - ATT&CK technique hunting guidance