Peak Threat Hunting V2

PEAK Threat Hunting in Splunk (V2 — Notable Events)

Guide threat hunts using the PEAK framework, executing SPL queries via the Splunk MCP server. This version adds automatic Notable Event creation in Splunk ES for hunt findings.

Quick Start

When a user requests a threat hunt:

1. Identify hunt type based on their goal:

   • Hypothesis-driven: Hunting specific adversary behavior (e.g., "Hunt for DNS tunneling")
   • Baseline: Understanding normal to find anomalies (e.g., "Baseline our firewall logs")
   • Model-Assisted (M-ATH): Using ML/statistical models (e.g., "Find anomalous login patterns")

2. Follow the PEAK phases: Prepare → Execute → Act

3. Run SPL queries using the Splunk MCP server's run_query tool (a logical name — see Splunk MCP Server Usage to resolve it to your server's actual tool)

4. Track progress using the hunt checklist for the selected type

5. Create Notable Events for confirmed findings (Act phase — requires ES)

Hunt Type Selection

Is this topic-based or data-based?
├── Topic-based (specific behavior/threat) → Does it have implicit complexity?
│   ├── Yes (many variables, ML needed) → Model-Assisted Hunt
│   └── No (explicit detection logic) → Hypothesis-Driven Hunt
└── Data-based (understand a data source) → Baseline Hunt

ES Availability Check

Before creating Notable Events, verify that Splunk Enterprise Security is installed.

Action: Run via Splunk MCP
Tool: run_query
Query: | rest /services/apps/local/SplunkEnterpriseSecuritySuite | table title, version
Purpose: Confirm ES is installed

• ES found: Proceed with Notable Event creation in the Act phase.
• ES not found: Warn the user that Notable Events cannot be created. Continue with standard findings presentation, dashboards, and documentation only.

Cache this result for the duration of the hunt session — do not re-check for every finding.

Hypothesis-Driven Hunt

Use when hunting for specific adversary behaviors or techniques.

Phase 1: Prepare

Hunt Preparation Checklist:
- [ ] Topic selected and scoped
- [ ] Research completed (TTPs, threat intel)
- [ ] Hypothesis generated (testable statement)
- [ ] ABLE framework applied
- [ ] Data sources identified
- [ ] Hunt plan created

1. Select Topic: Identify the behavior to hunt (e.g., "lateral movement", "credential theft")

2. Research Topic: Gather threat intelligence:

• Known techniques for implementing this tactic
• Existing detections and gaps
• Threat actor TTPs (check MITRE ATT&CK)
• Prior hunts on this topic

3. Generate Hypothesis: Create a testable, falsifiable statement:

Template: "[Actor] may be [behavior] in [location] using [technique]"
Example: "An adversary may be exfiltrating data via DNS tunneling from finance endpoints"

4. Apply ABLE Framework to scope the hunt:

5. Check Data Availability: Query available indexes and data:

Action: Run via Splunk MCP
Tool: list_indexes
Purpose: Identify available data sources

Action: Run via Splunk MCP
Tool: run_query
Query: | tstats count WHERE index=* by index, sourcetype | sort -count
Purpose: See data volume by sourcetype

Phase 2: Execute

Hunt Execution Checklist:
- [ ] Data gathered and filtered
- [ ] Data pre-processed if needed
- [ ] Analysis techniques applied
- [ ] Hypothesis refined (if needed)
- [ ] Critical findings escalated

1. Gather Data: Collect evidence based on ABLE scope

2. Pre-Process Data: Ensure data quality:

• Normalize timestamps to UTC
• Filter to relevant scope
• Handle missing values

3. Analyze: Apply hunting techniques:

4. Refine Hypothesis: If initial analysis is inconclusive, adjust scope or indicators

5. Escalate Critical Findings: Immediately escalate confirmed malicious activity

Phase 3: Act

Hunt Completion Checklist:
- [ ] Hunt preserved (queries, data, findings)
- [ ] Findings documented
- [ ] Notable Events created for confirmed findings (if ES available)
- [ ] Detections created/updated
- [ ] Dashboard created (if applicable)
- [ ] New hunt ideas logged
- [ ] Findings communicated to stakeholders

1. Preserve Hunt: Save queries, screenshots, and methodology

2. Document Findings: Use template in HUNT_TEMPLATES.md

3. Create Notable Events: Follow the Notable Event Creation Workflow below for each finding. Ask the user per-finding whether to create a Notable Event.

4. Create Detections: Convert findings to automated detection (see Detection Hierarchy below)

5. Create Dashboard: If the hunt produced findings that warrant ongoing monitoring or analyst review (Detection Hierarchy Level 2), invoke the $splunk-dashboard-studio skill. Pass it:

• All SPL queries used during the Execute phase (both successful and refined versions)
• The hunt hypothesis and ABLE scope (Actor, Behavior, Location, Evidence)
• Key findings: field names, notable values, thresholds discovered
• The time range used during the hunt
• Any severity classifications or risk scores identified
• Context about the target audience (SOC analysts, hunt team, management)

6. Communicate: Share with SOC, IR, detection engineering teams

Baseline Hunt

Use when establishing normal behavior for a data source to identify anomalies.

Phase 1: Prepare

Baseline Preparation Checklist:
- [ ] Data source selected
- [ ] Documentation/schema researched
- [ ] Scope defined (systems, timeframe)
- [ ] Analysis plan created

1. Select Data Source: Prioritize by security relevance or coverage gaps

2. Research Data Source: Understand field meanings and expected values:

Action: Run via Splunk MCP
Tool: run_query
Query: | metadata type=sourcetypes index=<target_index> | table sourcetype, totalCount, lastTime
Purpose: Understand sourcetype coverage

3. Scope: Define system groups and timeframe (typically 30-90 days)

Phase 2: Execute

Baseline Execution Checklist:
- [ ] Data gathered and filtered
- [ ] Data dictionary created
- [ ] Distributions reviewed
- [ ] Outliers investigated
- [ ] Gaps identified
- [ ] Relationships mapped

1. Create Data Dictionary: Document key fields:

Action: Run via Splunk MCP
Tool: run_query
Query: index=<target> | head 1000 | fieldsummary | table field, count, distinct_count, is_exact, numeric_count
Purpose: Understand field characteristics

2. Review Distributions: Establish baselines:

index=<target> | stats count by <field> | sort -count | head 20

index=<target> | stats avg(<field>) median(<field>) stdev(<field>) min(<field>) max(<field>)

index=<target> | stats dc(<field>) as unique_values

3. Investigate Outliers: Apply detection techniques from SPL_PATTERNS.md

4. Gap Analysis: Document missing data or quality issues

5. Identify Relationships: Map field correlations

Phase 3: Act

Document baseline including:

• Data dictionary
• Statistical summaries
• Known benign outliers
• Detection candidates

Create Notable Events: If the baseline hunt identified security-relevant outliers, follow the Notable Event Creation Workflow for each finding. Ask the user per-finding whether to create a Notable Event.

Create Baseline Dashboard: Baseline hunts are especially well suited for dashboards. Invoke the $splunk-dashboard-studio skill to create a baseline monitoring dashboard. Pass it:

• The distribution queries (categorical, numeric, cardinality)
• The data dictionary fields and their characteristics
• Outlier thresholds discovered during the hunt
• The target index, sourcetypes, and scope
• Time range for the baseline period The resulting dashboard gives analysts a reference view of "normal" to compare against during future investigations.

Model-Assisted Hunt (M-ATH)

Use when hunting requires ML/statistical models to find complex patterns.

Phase 1: Prepare

M-ATH Preparation Checklist:
- [ ] Topic selected with ML approach identified
- [ ] Research completed (methods, datasets)
- [ ] Datasets identified (labeled if supervised)
- [ ] Algorithm(s) selected

Algorithm Selection Guide:

Phase 2: Execute

M-ATH Execution Checklist:
- [ ] Data gathered and preprocessed
- [ ] Model developed and refined
- [ ] Model applied to hunt data
- [ ] Results analyzed
- [ ] Critical findings escalated

Key SPL for M-ATH:

index=<target> | anomalydetection <field> action=annotate

index=<target> | timechart count | predict count future_timespan=24

index=<target> | cluster field=<text_field> showcount=true

Phase 3: Act

Same as hypothesis-driven, but also:

• Preserve trained models
• Document model parameters and accuracy
• Consider model-based alerting
• Invoke the $splunk-dashboard-studio skill to create a model monitoring dashboard that tracks model outputs, anomaly scores, and prediction accuracy over time

Create Notable Events: Follow the Notable Event Creation Workflow for model-identified findings. Ask the user per-finding whether to create a Notable Event.

Notable Event Creation Workflow

This workflow creates immediate Notable Events in Splunk ES for hunt findings. It runs during the Act phase after findings are presented to the user.

Prerequisite: ES must be available (see ES Availability Check). If ES is not installed, skip this workflow entirely.

Step 1: Propose Notable Event Fields

For each finding, auto-derive the following fields and present them to the user for review:

Step 2: User Confirms or Adjusts

Present the proposed fields to the user and ask:

1. Whether to create a Notable Event for this finding
2. Whether any field values should be adjusted

Ask conversationally and wait for an explicit answer before proceeding.

Creating a Notable Event writes to Splunk ES and pages on-call downstream. Never create one without explicit user confirmation for that specific finding. If the user declines, skip to the next finding.

Step 3: Execute sendalert notable

Build and run the SPL query via the Splunk MCP:

Action: Run via Splunk MCP
Tool: run_query
Query:
| makeresults
| eval rule_name="<rule_name>",
       rule_description="<rule_description>",
       security_domain="<security_domain>",
       severity="<severity>",
       drilldown_search="<drilldown_search>",
       src="<src>",
       dest="<dest>",
       user="<user>",
       mitre_attack_id="<mitre_attack_id>"
| sendalert notable
Purpose: Create Notable Event for finding

• On success: Confirm to the user that the Notable Event was created, including the rule_name and severity.
• On error: Report the error to the user. Common issues: ES not installed, insufficient permissions, or the notable alert action not configured.

Step 4: Record in Hunt Documentation

Add the created Notable Event to the hunt report (see HUNT_TEMPLATES.md Notable Events section).

Detection Hierarchy

Convert hunt findings to automated detection. Choose the highest feasible level:

For Level 4 (Signatures/Rules): During this hunt, immediate Notable Events are created via | sendalert notable for confirmed findings. For persistent automated detection, recommend the user create a scheduled Correlation Search in ES using the hunt's SPL query.

For Level 2 (Dashboards): Always invoke the $splunk-dashboard-studio skill. It will take your hunt queries and findings and produce a validated dashboard with appropriate visualizations, drilldown, and filtering.

MITRE ATT&CK Integration

Map hunts to ATT&CK for coverage tracking. See MITRE_MAPPING.md for technique-specific hunting guidance.

Query ATT&CK-tagged events (if ES is deployed):

index=notable | stats count by mitre_attack_technique_id | sort -count

Hunt Metrics

Track these for each hunt:

See HUNT_TEMPLATES.md for metrics tracking template.

Splunk MCP Server Usage

Always use the Splunk MCP server to run queries (configured under mcp_servers in ~/.codex/config.toml). This skill refers to tools by logical name — different Splunk MCP servers expose them under different names, so at the start of a hunt, inspect the connected server's tool list and resolve each logical name to the actual tool. Do not guess: if no tool matches a capability, tell the user which one is missing.

Tool: run_query
Parameters:
  - query: SPL query string (required)
  - earliest_time: Start time (default: -24h)
  - latest_time: End time (default: now)
  - row_limit: Max results (default: 100, max: 1000)

Other useful tools:

• list_indexes: List available indexes
• list_knowledge_objects: List saved searches, alerts, macros, etc.

Notable Event creation pattern (requires ES):

Tool: run_query
Query: | makeresults | eval rule_name="...", severity="...", ... | sendalert notable
Purpose: Create an immediate Notable Event for a hunt finding

Additional Resources

• HUNT_TEMPLATES.md - Documentation templates
• SPL_PATTERNS.md - Common hunting SPL patterns
• MITRE_MAPPING.md - ATT&CK technique hunting guidance