First-run notice
Before doing any other work for this skill, follow references/first-run-notice.md exactly.
When you need to ask the user a question, get confirmation, or present choices, use the AskUserQuestion tool if available. This ensures proper rendering across all agent clients.
On-brand output
This skill's case-report HTML is styled from the shared Sysdig brand file (see the reporting templates) — never hand-craft styling. For anything else it renders itself, it carries the brand kit (get_brand_kit / get_design_tokens) — Path B of the shared methodology in references/on-brand-output.md.
It does not carry the governed author → validate → render tools, so don't call them here: if the user wants a shareable branded PDF instead of the HTML case report, hand off to the sysdig-report skill if it's installed, and otherwise say the PDF path isn't available in this skill.
Input
Two invocation forms:
/sysdig-runtime-investigate — interactive. The skill surfaces the top-priority threats and asks you to pick one.
/sysdig-runtime-investigate <event_id> — directed. The skill investigates the given event/threat directly.
Principles
Glossary. This skill uses these terms with specific meanings — keep them straight in user-facing text and in the case body:
- threat — a Threats Engine grouping (rule-level rollup of related events).
- event — a single security signal: Falco syscall, CloudTrail entry, or k8s_audit record.
- case — the report this skill produces and writes to
/tmp.
- incident — multiple correlated threats folded into the same case.
- image / workload / host — compute targets at three levels: image artifact, Kubernetes workload (Pod, Deployment, etc.), and underlying VM or bare-metal host.
These are the rules of the game. Phases are a floor, not a ceiling. The chosen threat is a starting point — the goal is the full attack chain, which often spans multiple Threats Engine groups, multiple resources, and events that don't appear correlated at first.
- Investigate freely. Phases are a floor; cross-cluster, cross-account, cross-threat-group correlation is expected when the signals support it.
- Do: fold related threat groups into the same case; pivot to CloudTrail when you see IMDS theft; expand the time window when an AWS access-key creation lines up with a K8s threat.
- Don't: stop at the first threat group; treat the trigger event as the whole story.
- Keep the user informed. Drop a one-line status update between non-trivial calls — what you just found, what you're doing next.
- Do: name the dimension you're pivoting to before the next call ("Process tree shows xmrig — looking for related cluster activity").
- Don't: emit silent multi-call blocks; batch tool calls without narration.
- Cite every claim. Every fact in the case body references its source — event ID, MCP (Model Context Protocol) tool name, REST path, or external URL.
- Do: attach the source inline for every CVE, IOC (indicator of compromise), process, and timestamp.
- Don't: paraphrase data without provenance; leave claims dangling.
- Don't fabricate. If a CVE, IOC, or process didn't appear in the data, don't write it.
- Do: omit fields the data didn't provide and record the limitation on the case object.
- Don't: invent details to fill gaps; infer beyond what the data supports.
- Two-tier output. The full case goes to a markdown file in
/tmp. The user sees a 2-paragraph summary in chat plus the file path.
- Do: keep tables, evidence dumps, and long traces in the file.
- Don't: paste tables into chat; force the user to scroll for the verdict.
- Read-only by design. This skill investigates and reports only. Remediation goes to
sysdig-runtime-remediate (for live response actions on the running threat) or sysdig-remediate (for image-level CVE fixes).
- Do: hand off to
sysdig-runtime-remediate when the user wants to act on the live threat (kill, isolate, pause), or to sysdig-remediate when the fix is a vulnerable image.
- Don't: run
kubectl delete, terraform apply, or any destructive operation directly.
- Always classify with MITRE. Assign one MITRE ATT&CK tactic to the trigger event — it drives the correlation gate and the watchlist mapping.
- Do: pick the more specific tactic when two could apply; record the secondary on
case.tactic_secondary.
- Don't: skip classification; defer it to a later phase.
- Always include
event.id. Surface it in the file footer and any handoff payload — it's the audit trail.
- Do: pass the event ID through Jira and PagerDuty payloads verbatim.
- Don't: redact, abbreviate, or drop the event ID anywhere downstream.
- Don't write cache or state to disk. The
/tmp report file is the only persistent artifact.
- Do: use MCP shared skill state for cross-skill context.
- Don't: write working state, raw event payloads, or secrets to disk.
State
State is read and written via the Sysdig MCP server tools.
| Operation |
Tool |
Arguments |
| Read state |
get_skill_state |
{ "skill_state": "runtime-investigate" } |
| Write state |
save_skill_state |
{ "skill_state": "runtime-investigate", "version": <n>, "data": { ... } } |
| Delete state |
delete_skill_state |
{ "skill_state": "runtime-investigate" } |
A null response from get_skill_state means no state exists yet — start with { "version": 0 }.
Schema
{
"version": 1,
"last_run": "2026-05-04T18:30:00Z",
"preferred_jira_project": "RUNTIME",
"preferred_handoff": "jira",
"recent_cases": [
{
"event_id": "abc123",
"cluster": "prod-eu-1",
"tactic": "c2",
"case_file": "/tmp/sysdig-runtime-investigate-abc123-20260504-1830.md",
"started": "2026-05-04T18:30:00Z",
"completed": "2026-05-04T18:42:00Z",
"handoff": { "destination": "jira", "ticket_key": "RUNTIME-1234", "ticket_url": "https://example.atlassian.net/browse/RUNTIME-1234" }
}
]
}
Read/write rules
- Get at the start of every session via
get_skill_state with { "skill_state": "runtime-investigate" }. A null response means no state — start with { "version": 0 }.
- Save at the end of every session via
save_skill_state with { "skill_state": "runtime-investigate", "version": <n>, "data": { ... } }. Read the current contents first, merge new data, then pass the full merged object as data.
- Version argument — the server uses
version for optimistic concurrency. Pass it as a separate argument (do not include it inside data):
- First write (
get_skill_state returned null) → call with version: 0.
- Subsequent writes → call with the same
version value the previous get_skill_state returned.
- On 409 conflict → call
get_skill_state again, merge your changes into the freshly-read state, retry once with the new version.
- Matching keys for upsert in
recent_cases: match on event_id. Cap the list at the 10 most recent entries; drop the oldest when over.
- Timestamps use ISO 8601. The
case_file path is informational only — files in /tmp may be cleaned up by the OS.
preferred_handoff is "jira" | "pagerduty" | "both" | "skip" | null. null means the user hasn't expressed a preference.
Steps
You run a 4-phase pipeline directly — no subagents.
Phase 0 ──→ Phase 1 ──→ Phase 2 ────→ Phase 3
Preflight Surface Investigate Synthesise + report
(free-form) (file + summary + handoff)
Phase 0 — Preflight
- Trust Preamble. Always present this before doing anything else. See
references/trust-preamble.md for the full text. After presenting the preamble, proceed directly to Step 0a — do NOT ask for confirmation. The preamble is informational.
0a. MCP authentication preflight. Before any other step — including the state read below — run the preflight in references/auth-preflight.md and follow its instructions exactly. This skill requires the Sysdig MCP: it is the only source for Threats Engine, runtime events, process trees, SysQL, vulnerability scans, and the threat-intel feed. If the preflight returns State 2 (registered but not authenticated) or State 3 (not reachable), emit its verbatim message and stop — no data calls, no state read, no file writes. Do NOT call mcp__secure-mcp-server__authenticate autonomously. Only continue past this step on State 1 (catalog reachable).
0b. Read shared state. Call mcp__secure-mcp-server__get_skill_state with { "skill_state": "runtime-investigate" }. A null response means no prior state — start with { "version": 0 }. If state exists, hydrate preferred_jira_project, preferred_handoff, and recent_cases onto the working session so Phase 3 can default to the user's prior choices instead of re-asking.
Reporting / CTI (cyber threat intelligence) probe (no-block). Detect available destinations and CTI tools dynamically. Do not require any specific env-var name — match by pattern.
- Jira / case tracking: scan for MCP tools matching
mcp__atlassian__*, mcp__*jira*, mcp__*linear*. Mark the first match.
- Jira project key: scan env vars matching
*JIRA_PROJECT* (e.g. SYSDIG_RUNTIME_JIRA_PROJECT). If matched, surface the value as the default project for Phase 3 handoff.
- PagerDuty / on-call: scan env vars matching
*PAGERDUTY*, *PD_TOKEN*, *PD_ROUTING_KEY*.
- VirusTotal: scan env vars matching
*VIRUSTOTAL*, *VT_API*, *VT_KEY*.
Record what was found. Do not prompt yet if nothing was detected — defer that to Phase 3 handoff.
Announce connectivity. Before moving on, surface a single user-facing line summarising what's wired up. Mark each integration ✓ (detected), — (missing), or ✗ (probe failed). The Sysdig MCP is always ✓ here — Step 0a already guaranteed it. Example:
Connectivity: Sysdig MCP ✓ · Jira ✓ · PagerDuty — · VirusTotal —. Will skip binary VT enrichment and PagerDuty handoff.
Data-egress notice. Immediately after the connectivity line, when any third-party enrichment is reachable, disclose in one line what indicators will leave the environment and to which service — so the user can veto before any external call:
Enrichment will send: binary SHA256 hashes → VirusTotal · public (non-private) IPs → ipinfo.io (fallback ip-api.com) · CVE IDs → NVD/CISA KEV/Exploit-DB/GHSA. Private/internal IPs (RFC1918, link-local, loopback, CGNAT) are never sent. No Sysdig event contents, hostnames, or Sysdig credentials leave the environment — the VirusTotal call carries only the hash plus your VirusTotal API key.
Only list the destinations that are actually reachable (✓). If the user declines, skip the corresponding enrichment and record it as skipped: user declined egress. This is the only narration during Phase 0 — the per-step results above are recorded silently on the case object. Don't prompt beyond the egress veto; just inform.
Entry-point detection. Parse the invocation argument:
- No argument →
interactive mode.
- Anything else →
directed mode, store the value as the event ID.
Resume check (directed mode only). If recent_cases (from step 0b) has an entry whose event_id matches the current trigger, surface a one-line resume summary and ask the user via AskUserQuestion how to proceed:
"You investigated <event_id> hours ago — case file <case_file>, handoff <destination>:<key> if any. Continue with the saved case, refresh (re-run Phase 2), or start fresh (clear the prior entry)?"
- Continue → skip Phase 1 + Phase 2; jump to Phase 3 with the saved case (read the file from disk if it still exists; otherwise treat as
refresh).
- Refresh → keep the saved entry but re-investigate; merge new findings.
- Start fresh → drop the prior entry from
recent_cases and re-run.
Staleness default. If the prior completed timestamp is more than 4 hours ago, runtime data has likely shifted — pre-select Refresh as the default. Below 4 hours, default to Continue.
Interactive mode does not pause here — Phase 1 will surface fresh threats and any prior case will only show up in the picker as (seen <N>h ago).
Phase 1 — Surface
Interactive flow:
List the top open threats — Threats Engine is exposed via MCP. Call mcp__secure-mcp-server__list_threats_engine_groups with limit=5, sorted by lastSignal:desc, status=open.
If Threats Engine returns empty (unavailable or quiet in this tenant), fall back to runtime events for the same window: mcp__secure-mcp-server__list_runtime_events with last 24h, limit 10.
Present the result as a markdown table:
| # |
Severity |
Rule / aiGeneratedName |
Resource |
Last seen |
Ask via AskUserQuestion: "Which one do you want to investigate?"
Incident-scope detection at surface time. Before diving into the chosen threat, scan the other surfaced groups. Multi-stage attacks frequently span more than one Threats Engine grouping. Treat all groups sharing cluster + ±2h, OR aws.accountId + ±2h, OR same image as the same incident — one investigation, one case body, one narrative. Record them on case.incident_threat_groups.
If these conditions hold, the chosen threat is one facet of a larger incident — fold all matching groups into the same case object. Tag them on case.incident_threat_groups (id, name, resource, last_seen, why_related). Phase 2's cluster-wide sweep then has a head start — these groups' constituent events should also appear in the sweep, but flagging them upfront lets the report's "Incident scope" section name them by their AI-generated title rather than as anonymous events.
Directed flow:
- Resolve the specific threat. The trigger ID may be a threat ID, a group ID, or an event ID — try in that order:
mcp__secure-mcp-server__get_threats_engine_threat (single-threat detail); on 404 mcp__secure-mcp-server__get_threats_engine_group, then mcp__secure-mcp-server__list_threats_engine_threats_by_group with the ID; if all 404, fall back to mcp__secure-mcp-server__get_event_info with the event ID.
Classification — MITRE ATT&CK tactic. From the rule name, rule source, and event labels, assign one MITRE tactic to the threat. Store it on the case as case.tactic. Phase 2 watchlist mapping reads this value. See references/mitre-tactics.md for the keyword-to-tactic table and the secondary-tactic rule.
Process tree (Sysdig MCP). If the threat has an event_id (the Threats Engine returns securityEvent references with IDs), call mcp__secure-mcp-server__get_event_process_tree with the event ID to retrieve the structured process tree. Store the parsed result (parent → child chain, command lines, sha256 if present) on case.process_tree.
Process evidence from aiGeneratedDescription (always runs — useful even alongside the structured tree). The description carries natural-language context the structured tree doesn't ("locale repeatedly", "curl --upload to external IP"). Parse it for process names (e.g. systemd, sshd, bash, curl, wget, nc, nslookup) and chain hints ("spawned by", "child of"). Store as case.process_evidence (list of strings).
The two are complementary: case.process_tree is structured ground truth, case.process_evidence is the AI's narrative read of the same chain. The report renders both in "What happened" — the tree as a tree, the evidence as a one-liner.
Store the threat, classification, secondary tactic (if any), process tree (if available), and process evidence on the working case object.
Phase 2 — Investigate (free-form, signal-driven)
Goal: reconstruct the full attack chain starting from the user's pick. Span multiple threat groups, multiple resources, multiple event sources if the signals lead there. The chain is the deliverable — phase boundaries from earlier versions of this skill (e.g. separate enrichment / classifier / synthesis stages) are explicitly not prescribed steps anymore.
Tell the user what you're doing as you go. Examples of good status updates:
- "Process tree shows Tomcat → bash → xmrig — looks like miner persistence. Looking for related cluster activity."
- "IMDS (Instance Metadata Service) theft on the host. Expanding to CloudTrail in the same AWS account."
- "Found two more threat groups on the same account in the same hour — folding them in as the same campaign."
- "Runtime scan for the workload came back empty — pulling image-level vuln findings instead."
Available signals (chase them when relevant)
These are the ingredients. The order is yours.
Time scoping — read before any event query. list_runtime_events, count_runtime_events, and discover_runtime_event_field_values accept only scope_hours — a window counted back from now. There is no absolute start/end, and filter_expr has no timestamp field. So a "±2h around the trigger" or "last 7 days" window is not something scope_hours gives you when the trigger isn't recent: set scope_hours wide enough to reach past the trigger (scope_hours ≈ hours_since_trigger + 2), then post-filter the returned events by their timestamp down to the band you actually want. Treat every "±2h" / "7-day" phrasing below as the target band after client-side filtering, not a parameter you can pass directly.
Tenant-wide critical sweep (sanity check) — once early in Phase 2, call mcp__secure-mcp-server__list_runtime_events with filter_expr = "severity in (0,1,2,3)" and no scope filter across the ±2h window around the trigger. Catches cross-domain signals the cluster/account-filtered queries miss (GitHub cloudProvider.account.id, Okta cloudProvider.tenantId, anything without K8s labels). Fold in any hit whose image-org, repo name, or actor matches the trigger.
Process tree of the trigger event — mcp__secure-mcp-server__get_event_process_tree. Almost always the highest-yield single artifact. Falls back to aiGeneratedDescription parsing if the MCP returns empty.
Activity audit trail of the affected workload — the agent-captured record of every command, network connection, and file access around the trigger, independent of whether each tripped a rule. This is what a human analyst pulls in the Secure UI to answer "what else did this workload do around the detection?" — dwell time, lateral-movement setup, dropped files, C2 callbacks. Use the MCP tool list_activity_audit_events scoped to the container / pod / workload (never the bare host) and centred on the trigger time. Unlike list_runtime_events, the activity audit tools take a real absolute window (from/to as ISO-8601, epoch, or a relative offset like 2h/7d, max 2 weeks) — so a "±2h around the trigger" band is a direct parameter, not the scope_hours post-filter workaround. For a suspicious command (dropper, reverse shell, miner), pass its entry id to trace_activity_audit_command to reconstruct the child processes, connections, and files it spawned in the same session — the audit-trail counterpart to the process tree. See references/activity-audit.md for tools, filter fields, scoping, and the timeline output shape. The trail feeds the "Activity audit timeline" report section and cross-references into the network/binary IOC enrichment.
Prior events on the affected resource — last 7 days, via mcp__secure-mcp-server__list_runtime_events with a filter_expr matching the workload (kubernetes.cluster.name + namespace + workload) or host (host.hostName). For K8s workloads, also pull host-level events on the same node — escapes hide there.
Cluster-wide activity in a ±2h window around the trigger. Same MCP tool, three filters in parallel. The source values are tenant/region/backend-specific — confirm the real ones with mcp__secure-mcp-server__discover_runtime_event_field_values on the source field before filtering, rather than assuming a literal. Known values at the time of writing: syscall, k8s_audit, awscloudtrail + awscloudtrail_stateful (AWS — note there is no bare cloudtrail value), azure_platformlogs + azure_entra (Azure), gcp_auditlog (GCP). If a source filter comes back empty, re-run discover (its suggested bucket lists what's actually firing, other lists what the tenant knows) and widen to the values it reports.
kubernetes.cluster.name = "<cluster>" and source = "syscall" (other resources in the cluster)
kubernetes.cluster.name = "<cluster>" and source = "k8s_audit" (Attach/Exec Pod, Deployment Created, etc.)
kubernetes.cluster.name = "<cluster>" and source in ("awscloudtrail", "awscloudtrail_stateful") (cluster-tagged cloud events, if any — swap in the Azure/GCP source values from discover when the cluster runs there)
Cloud-account-wide activity when the resource has aws.accountId / azure.subscriptionId / gcp.projectId. Cloud audit-log and agentless-ML events live under the account dimension, not the cluster. This is the difference between catching multi-stage cross-cloud attacks (IMDS credential theft → IAM access-key creation → audit-log tampering → storage exfiltration) and missing them. Filter by the matching dimension and that cloud's source values (verify with discover first):
- AWS —
aws.accountId = "<account>" and source in ("awscloudtrail", "awscloudtrail_stateful", "agentless-aws-ml")
- Azure —
azure.subscriptionId = "<subscription>" and source in ("azure_platformlogs", "azure_entra")
- GCP —
gcp.projectId = "<project>" and source in ("gcp_auditlog")
Other threat groups in this incident — fold them into the same case — do not investigate separately. Other threat groups already tagged in Phase 1 as part of this incident — pull their constituent events via mcp__secure-mcp-server__list_threats_engine_threats_by_group and their affected resources via mcp__secure-mcp-server__list_threats_engine_resources_by_group (both keyed by the group ID) to merge into the chain. If new groups appear in the cluster window during Phase 2 investigation, fold them in too. Cross-type is allowed: a CLOUD threat may be the same incident as a K8S_WORKLOAD threat.
Sibling resources / posture / RBAC (role-based access control) via mcp__secure-mcp-server__run_sysql. SysQL schema differs between tenants — adjust query shape if rejected. Example queries: MATCH KubeWorkload AS wl WHERE wl.cluster = '<c>' RETURN wl.namespace, wl.name, MATCH Resource VIOLATES Control, MATCH KubeServiceAccount HAS KubeRoleBinding HAS KubeClusterRole.
Vulnerability surface — start with mcp__secure-mcp-server__query_vulnerability_findings scoped to the running image, filtered to the notable CVEs:
{
"group_by": "cve",
"filters": [
{ "field": "imageId", "op": "eq", "value": "<running image's config digest>" },
{ "field": "severity", "op": "in", "value": ["critical", "high"] },
{ "field": "hasFix", "op": "eq", "value": true },
{ "field": "hasExploit", "op": "eq", "value": true }
]
}
Scope by the right identifier. imageId expects the image config digest — the sha256:… that runtime detection resolves the running container to (the threat detail's image id; equivalently metadata.imageId from get_scan_result in the scan-result step below). This is a different sha256 from the registry manifest digest (findingsapi field imageDigest, metadata.digest) — cross them and the query returns zero rows with no error. If you only have the pullstring, use imageReference (e.g. …/app:1.0.0) instead, accepting that a mutable tag may span builds.
Filtering. The tool applies no default filters, so set the triage filters explicitly: severity critical/high, hasFix true, hasExploit true. That narrows to the actionable, weaponized CVEs the MITRE-tactic gate and CTI step below consume. If the result comes back empty, widen by dropping hasExploit, then hasFix, before concluding the image is clean.
group_by: "cve" returns one deduplicated row per CVE, ranked by the grouping's default risk order. Read rows from the response's data array; meta.hasMore + the cursor token page further, and severity comes back capitalized (Critical) in responses even though the filter value is lowercase. For running/in-use package detail — which vulnerable packages are actually loaded on the workload or host — use mcp__secure-mcp-server__list_runtime_scan_results (filter by kubernetes.cluster.name/namespace/workload or host.hostName, sort runningVulnTotalBySeverity) and pass the returned resultId to mcp__secure-mcp-server__get_scan_result for the full packages + vulnerabilities breakdown.
External CTI for the top 5 critical/high CVEs that pass the MITRE-tactic gate (see references/correlation-guide.md): NVD, CISA KEV (Known Exploited Vulnerabilities), Exploit-DB, GHSA (GitHub Security Advisory) via WebFetch. Don't fetch CTI for tactic-mismatched CVEs.
Sysdig threat-intel feed via mcp__secure-mcp-server__fetch_threat_intelligence_feed — Sysdig-curated CVEs / zero-days / active-attack notes. Cross-reference any IOCs you collect.
VirusTotal (VT) for binary IOCs when a SHA256 surfaces on event fields and a VT key is present (Phase 0 records the env var). When the threat lacks proc.sha256, look across other events on the same container — drift detection events typically carry the hash.
GeoIP for network IOCs via curl https://ipinfo.io/<ip>/json (fallback ip-api.com).
Patterns to recognise (not steps to execute)
When any of these rule names fire in the data you've pulled, that's the signal to expand. The action column is what direction to look, not a fixed query. See references/watchlist-patterns.md for the full hit-to-pivot table covering IMDS theft, persistent cloud creds, defense evasion, S3 exfil, privesc, lateral movement, runtime compromise, recon, and prompt-injection.
MITRE-tactic correlation
Read references/correlation-guide.md for the MITRE-tactic gate and 1–5 scoring heuristics. Render only pairs ≥ 4 in the case body. Boosts (cap at 5): KEV +1, VT malicious_count≥5 +1.
Stop conditions
You're done when:
- The chain has a coherent narrative (initial access → execution → … → impact, or bounded by available data), AND
- You've followed every watchlist hit at least one hop, AND
- You've pulled the activity audit trail for the affected container/pod/workload around the trigger (or recorded why it was unavailable) — the timeline is part of the deliverable, AND
- You've asked yourself "is there another threat group within the cluster + ±2h / account + ±2h window that belongs to this same incident?" — and either folded it in or recorded why not.
If you find yourself making more than ~4 follow-on calls per signal, stop and synthesise. Diminishing returns.
Phase 3 — Synthesise & report
Four steps, in this order:
Ask the user for the report format via AskUserQuestion: "How do you want the report rendered? Markdown (default — plays nice with Jira/PagerDuty handoff) / HTML (renders Mermaid diagrams in browser, self-contained) / Both."
If AskUserQuestion is not available, default to Markdown and mention HTML as a follow-up option in the summary.
Write the file(s) to disk. Base path: /tmp/sysdig-runtime-investigate-<event_id_short>-<UTC-yyyymmdd-hhmm>.
Markdown → write <base>.md with Block 1 of references/reporting-templates.md verbatim. Always written when chosen — also the body of Jira/PagerDuty handoff downstream.
HTML → write <base>.html using Block 4 wrapper from the same file. The Block 1 markdown content is pasted into the wrapper's <script id="md"> block. Marked + mermaid CDN scripts render the page in the browser.
Both → write both files, same base name.
Print a 2-paragraph summary to the user. Use Block 0 of reporting-templates.md — short, no tables. Cite the file path(s) so they can read it if they want. Then ask AskUserQuestion: "Where do you want to report this case?" with options derived from what Phase 0 detected (Jira, PagerDuty, Both, Just show it).
Handoff:
Jira → Build the Block 2 payload from the templates (body = file content). If the project key is unknown, call mcp__atlassian__getVisibleJiraProjects and ask the user. Preview before submitting. Render a short block — Project · Issue type · Summary · Severity · Labels · Body length (chars) — then ask via AskUserQuestion: "Submit, edit, or cancel?". Only on Submit call mcp__atlassian__createJiraIssue. Capture key + URL and surface them with an undo line: "Undo: open the ticket and click Close."
PagerDuty → Build the Block 3 payload (custom_details.case = file content, truncate to ~30 KB if larger). Preview before submitting. Render Routing key (last 4) · Severity · Dedup key · Summary · Source · custom_details size (KB) and ask via AskUserQuestion: "Submit, edit, or cancel?". Only on Submit send via:
printf '%s' "$PAYLOAD_JSON" | curl -sS -X POST \
-H "Content-Type: application/json" \
--data-binary @- \
https://events.pagerduty.com/v2/enqueue
Capture the dedup_key and incident URL from the response. Surface them with an undo line: "Undo: resolve the incident with event_action: resolve and the same dedup key."
Both → run the Jira preview-and-submit first, then the PagerDuty preview-and-submit. Confirm each separately; declining one does not cancel the other.
Just show → no extra action; the file path was already cited in the summary.
If the user picks Edit at the preview step, ask which fields to change, apply the edits to the payload, and re-show the preview. Loop until the user picks Submit or Cancel.
After handoff, surface the link/key (if any) to the user — one short line.
Offer to act on the threat (optional handoff to remediation). After the reporting handoff is complete (or skipped), ask via AskUserQuestion: "Want me to act on this threat now? sysdig-runtime-remediate will propose response actions (kill, isolate, pause, etc.), explain the consequences on the live workload, and execute only what you approve." Options: Yes — invoke sysdig-runtime-remediate / No.
- On Yes — announce the handoff explicitly per principle C-03 ("Handing off to
sysdig-runtime-remediate with <event_id>. The case file you just generated will be its starting context."), then invoke sysdig-runtime-remediate <event_id>.
- On No — proceed silently to step 5. The user can always come back later with
/sysdig-runtime-remediate <event_id> — the case file and shared state will still be there.
Skip this step if sysdig-runtime-remediate is not registered in the current plugin (back-compat for environments running an older plugin version).
Persist shared state. Before exiting, call mcp__secure-mcp-server__save_skill_state with { "skill_state": "runtime-investigate", "version": <version-from-phase-0>, "data": { ... } }. Merge the freshly-read state from Phase 0 with the new entries: append this case to recent_cases (upsert by event_id, cap at 10), update last_run, and update preferred_jira_project / preferred_handoff if the user picked a destination. Skip silently if the Sysdig MCP isn't loaded.
Error Handling
Status vocabulary. Every section of the case object uses the house status set: done, pending, in_progress, failed, skipped. The reason goes in a separate reason field so the report and the summary can render either status alone or status + reason.
{ "vulnerabilities": { "status": "skipped", "reason": "no scan data" } }
{ "cloudtrail": { "status": "skipped", "reason": "no integration in tenant" } }
{ "vt": { "status": "skipped", "reason": "no API key" } }
{ "cti.nvd": { "status": "failed", "reason": "rate-limited" } }
User-facing error template. Every error message the user sees follows three lines: what failed, why, and the fix (a copy-pasteable command or concrete next step). Keep the whole thing under four lines. Examples:
Can't reach Sysdig — the MCP server isn't registered or authenticated.
Register it with claude mcp add --transport http secure-mcp-server https://<region>/mcp/secure, then run /mcp → Sysdig → Authenticate.
Per-region URLs are in references/mcp-setup.md. Then re-run.
No threats matched the last 24h.
Either the tenant is quiet or the time window is too tight.
Widen the time window (e.g. 7 days) or pick a specific event ID.
Couldn't open the Jira ticket — Atlassian MCP returned 403.
Your token doesn't have write:jira-work for project RUNTIME.
Refresh the token in the Atlassian admin console and retry, or pick another destination.
Truncation, skipped enrichments, and other partial-success cases must be flagged in the 2-paragraph chat summary too — never let the user discover them only by reading the case file.
| Situation |
Behavior |
| Sysdig MCP preflight returns State 2 or State 3 |
Abort in Phase 0a per auth-preflight.md — emit its verbatim message and stop. No data calls, no state read, no file writes. |
| An MCP tool call comes back denied (the agent refuses the call rather than the server erroring) |
Not an auth or reachability failure — the tool isn't on the agent's allowlist, usually a prefix mismatch. allowed-tools lists the canonical mcp__secure-mcp-server__* names (per mcp-setup.md), but the server may be registered under a different prefix in this agent (e.g. a plugin-managed name). The bare tool names match 1:1; only the prefix differs. Tell the user: confirm the registered name with claude mcp list, then either re-register it as secure-mcp-server (the mandated name — see mcp-setup.md) or add the actual mcp__<prefix>__* pattern to the agent's permitted tools, and re-run. Do not loop retrying the denied call. |
Trigger ID resolves to neither a threat nor a group (get_threats_engine_threat, get_threats_engine_group, and list_threats_engine_threats_by_group all 404) |
Fall back to mcp__secure-mcp-server__get_event_info with the same ID. If that 404s too, tell the user the ID wasn't found and offer interactive mode. |
| No qualifying threats / events found in Phase 1 |
Tell the user, offer to widen the time window. |
| Vulnerability lookup empty |
When mcp__secure-mcp-server__query_vulnerability_findings returns an empty data array, widen by relaxing filters (drop hasExploit, then hasFix, then severity) before concluding the image is clean. If still empty, try mcp__secure-mcp-server__list_runtime_scan_results (filter by workload/host) then mcp__secure-mcp-server__get_scan_result. If still nothing, record { "vulnerabilities": { "status": "skipped", "reason": "no scan data" } } and continue. |
WebFetch to a public CTI source fails or rate-limits |
Mark that source { "status": "failed", "reason": "<rate-limited | network | 4xx>" } and continue. Don't retry in a loop. |
| VirusTotal API key not detected |
Record { "vt": { "status": "skipped", "reason": "no API key" } }. |
| No Jira and no PagerDuty detected |
Ask whether the user wants to configure one or just show the file path. |
| Atlassian MCP / PagerDuty curl errors during handoff |
Record { "handoff": { "status": "failed", "reason": "<error>" } }, surface the error, print the file path, offer to retry or switch destination. |
| Activity audit trail empty or capture not enabled |
list_activity_audit_events returns no entries for the scoped container/pod/workload. Not a failure — the agent's command/connection/file capture may be off, or the workload was genuinely quiet. Record { "activity_audit": { "status": "skipped", "reason": "no activity audit data for this scope" } } and flag in the summary that audit context was unavailable. |
| Activity audit window exceeds 2 weeks |
from/to span is capped at 2 weeks server-side. Narrow the window (centre tighter on the trigger) and re-issue. |
mcp__secure-mcp-server__list_runtime_events cursor pagination drops filter_expr |
Known MCP quirk — paginating with cursor returns events from the wrong scope. Don't paginate; widen scope_hours and re-issue with the original filter. |
| A list_runtime_events call returns >100 events |
Tighten the time window or split the filter. Never silently drop the section — record { "status": "done", "truncated": true, "shown": <n>, "total": <m> }. |
| Cloud audit-log integration (CloudTrail / GCP Audit / Azure Activity) not present in tenant |
Cloud-account sweep returns empty (no source values for that provider — confirm with discover). Record { "cloudtrail": { "status": "skipped", "reason": "no integration in tenant" } } on the case; flag in the summary that cloud-API signals (storage / IAM / IMDS) were unavailable. |
| SysQL schema rejects a query |
Try the alternate relation shapes (HAS vs BINDS_TO, etc.). If all fail, record { "status": "failed", "reason": "schema mismatch" } and continue. The skill should not block on schema mismatches. |
1---2name: sysdig-runtime-investigate3description: Use this skill when investigating a runtime threat detected by Sysdig end-to-end. Surfaces the highest-priority threat, scores vulnerability vs runtime correlations on a 1-5 confidence scale, deep-dives into network blast radius or suspicious-binary VirusTotal lookups depending on the event class, reconstructs the affected workload's activity audit trail (commands, connections, file accesses) as a timeline around the detection, and hands the case off to Jira or PagerDuty. Triggers on: "investigate runtime threat", "what is this Falco alert", "triage this SOC alert", "analyze runtime incident". Not for vulnerability prioritization (use `sysdig-investigate`) or remediation (use `sysdig-remediate`).4---56## First-run notice78Before doing any other work for this skill, follow [`references/first-run-notice.md`](references/first-run-notice.md) exactly.91011When you need to ask the user a question, get confirmation, or present choices, use the `AskUserQuestion` tool if available. This ensures proper rendering across all agent clients.1213## On-brand output1415This skill's case-report HTML is styled from the shared Sysdig brand file (see the reporting templates) — never hand-craft styling. For anything else it renders itself, it carries the brand kit (`get_brand_kit` / `get_design_tokens`) — **Path B** of the shared methodology in [`references/on-brand-output.md`](references/on-brand-output.md).1617It does **not** carry the governed author → validate → render tools, so don't call them here: if the user wants a shareable branded **PDF** instead of the HTML case report, hand off to the `sysdig-report` skill if it's installed, and otherwise say the PDF path isn't available in this skill.1819## Input2021Two invocation forms:2223- `/sysdig-runtime-investigate` — interactive. The skill surfaces the top-priority threats and asks you to pick one.24- `/sysdig-runtime-investigate <event_id>` — directed. The skill investigates the given event/threat directly.2526## Principles2728**Glossary.** This skill uses these terms with specific meanings — keep them straight in user-facing text and in the case body:2930- **threat** — a Threats Engine grouping (rule-level rollup of related events).31- **event** — a single security signal: Falco syscall, CloudTrail entry, or k8s_audit record.32- **case** — the report this skill produces and writes to `/tmp`.33- **incident** — multiple correlated threats folded into the same case.34- **image / workload / host** — compute targets at three levels: image artifact, Kubernetes workload (Pod, Deployment, etc.), and underlying VM or bare-metal host.3536These are the rules of the game. **Phases are a floor, not a ceiling.** The chosen threat is a starting point — the goal is the full attack chain, which often spans multiple Threats Engine groups, multiple resources, and events that don't appear correlated at first.3738- **Investigate freely.** Phases are a floor; cross-cluster, cross-account, cross-threat-group correlation is expected when the signals support it.39 - Do: fold related threat groups into the same case; pivot to CloudTrail when you see IMDS theft; expand the time window when an AWS access-key creation lines up with a K8s threat.40 - Don't: stop at the first threat group; treat the trigger event as the whole story.41- **Keep the user informed.** Drop a one-line status update between non-trivial calls — what you just found, what you're doing next.42 - Do: name the dimension you're pivoting to before the next call ("Process tree shows xmrig — looking for related cluster activity").43 - Don't: emit silent multi-call blocks; batch tool calls without narration.44- **Cite every claim.** Every fact in the case body references its source — event ID, MCP (Model Context Protocol) tool name, REST path, or external URL.45 - Do: attach the source inline for every CVE, IOC (indicator of compromise), process, and timestamp.46 - Don't: paraphrase data without provenance; leave claims dangling.47- **Don't fabricate.** If a CVE, IOC, or process didn't appear in the data, don't write it.48 - Do: omit fields the data didn't provide and record the limitation on the case object.49 - Don't: invent details to fill gaps; infer beyond what the data supports.50- **Two-tier output.** The full case goes to a markdown file in `/tmp`. The user sees a 2-paragraph summary in chat plus the file path.51 - Do: keep tables, evidence dumps, and long traces in the file.52 - Don't: paste tables into chat; force the user to scroll for the verdict.53- **Read-only by design.** This skill investigates and reports only. Remediation goes to `sysdig-runtime-remediate` (for live response actions on the running threat) or `sysdig-remediate` (for image-level CVE fixes).54 - Do: hand off to `sysdig-runtime-remediate` when the user wants to act on the live threat (kill, isolate, pause), or to `sysdig-remediate` when the fix is a vulnerable image.55 - Don't: run `kubectl delete`, `terraform apply`, or any destructive operation directly.56- **Always classify with MITRE.** Assign one MITRE ATT&CK tactic to the trigger event — it drives the correlation gate and the watchlist mapping.57 - Do: pick the more specific tactic when two could apply; record the secondary on `case.tactic_secondary`.58 - Don't: skip classification; defer it to a later phase.59- **Always include `event.id`.** Surface it in the file footer and any handoff payload — it's the audit trail.60 - Do: pass the event ID through Jira and PagerDuty payloads verbatim.61 - Don't: redact, abbreviate, or drop the event ID anywhere downstream.62- **Don't write cache or state to disk.** The `/tmp` report file is the only persistent artifact.63 - Do: use MCP shared skill state for cross-skill context.64 - Don't: write working state, raw event payloads, or secrets to disk.6566## State6768State is read and written via the Sysdig MCP server tools.6970| Operation | Tool | Arguments |71|-----------|------|-----------|72| Read state | `get_skill_state` | `{ "skill_state": "runtime-investigate" }` |73| Write state | `save_skill_state` | `{ "skill_state": "runtime-investigate", "version": <n>, "data": { ... } }` |74| Delete state | `delete_skill_state` | `{ "skill_state": "runtime-investigate" }` |7576A `null` response from `get_skill_state` means no state exists yet — start with `{ "version": 0 }`.7778### Schema7980```json81{82 "version": 1,83 "last_run": "2026-05-04T18:30:00Z",84 "preferred_jira_project": "RUNTIME",85 "preferred_handoff": "jira",86 "recent_cases": [87 {88 "event_id": "abc123",89 "cluster": "prod-eu-1",90 "tactic": "c2",91 "case_file": "/tmp/sysdig-runtime-investigate-abc123-20260504-1830.md",92 "started": "2026-05-04T18:30:00Z",93 "completed": "2026-05-04T18:42:00Z",94 "handoff": { "destination": "jira", "ticket_key": "RUNTIME-1234", "ticket_url": "https://example.atlassian.net/browse/RUNTIME-1234" }95 }96 ]97}98```99100### Read/write rules101102- **Get** at the start of every session via `get_skill_state` with `{ "skill_state": "runtime-investigate" }`. A `null` response means no state — start with `{ "version": 0 }`.103- **Save** at the end of every session via `save_skill_state` with `{ "skill_state": "runtime-investigate", "version": <n>, "data": { ... } }`. Read the current contents first, merge new data, then pass the full merged object as `data`.104- **Version argument** — the server uses `version` for optimistic concurrency. Pass it as a separate argument (do not include it inside `data`):105 - First write (`get_skill_state` returned `null`) → call with `version: 0`.106 - Subsequent writes → call with the same `version` value the previous `get_skill_state` returned.107 - On 409 conflict → call `get_skill_state` again, merge your changes into the freshly-read state, retry once with the new version.108- **Matching keys** for upsert in `recent_cases`: match on `event_id`. Cap the list at the 10 most recent entries; drop the oldest when over.109- Timestamps use ISO 8601. The `case_file` path is informational only — files in `/tmp` may be cleaned up by the OS.110- `preferred_handoff` is `"jira" | "pagerduty" | "both" | "skip" | null`. `null` means the user hasn't expressed a preference.111112## Steps113114You run a 4-phase pipeline directly — no subagents.115116```117Phase 0 ──→ Phase 1 ──→ Phase 2 ────→ Phase 3118Preflight Surface Investigate Synthesise + report119 (free-form) (file + summary + handoff)120```121122### Phase 0 — Preflight1231240. **Trust Preamble.** Always present this before doing anything else. See [`references/trust-preamble.md`](references/trust-preamble.md) for the full text. After presenting the preamble, proceed directly to Step 0a — do NOT ask for confirmation. The preamble is informational.1251260a. **MCP authentication preflight.** Before any other step — including the state read below — run the preflight in [`references/auth-preflight.md`](references/auth-preflight.md) and follow its instructions exactly. This skill **requires** the Sysdig MCP: it is the only source for Threats Engine, runtime events, process trees, SysQL, vulnerability scans, and the threat-intel feed. If the preflight returns State 2 (registered but not authenticated) or State 3 (not reachable), emit its verbatim message and **stop** — no data calls, no state read, no file writes. Do NOT call `mcp__secure-mcp-server__authenticate` autonomously. Only continue past this step on State 1 (catalog reachable).1271280b. **Read shared state.** Call `mcp__secure-mcp-server__get_skill_state` with `{ "skill_state": "runtime-investigate" }`. A `null` response means no prior state — start with `{ "version": 0 }`. If state exists, hydrate `preferred_jira_project`, `preferred_handoff`, and `recent_cases` onto the working session so Phase 3 can default to the user's prior choices instead of re-asking.1291301. **Reporting / CTI (cyber threat intelligence) probe (no-block).** Detect available destinations and CTI tools dynamically. Do not require any specific env-var name — match by pattern.131132 - Jira / case tracking: scan for MCP tools matching `mcp__atlassian__*`, `mcp__*jira*`, `mcp__*linear*`. Mark the first match.133 - Jira project key: scan env vars matching `*JIRA_PROJECT*` (e.g. `SYSDIG_RUNTIME_JIRA_PROJECT`). If matched, surface the value as the default project for Phase 3 handoff.134 - PagerDuty / on-call: scan env vars matching `*PAGERDUTY*`, `*PD_TOKEN*`, `*PD_ROUTING_KEY*`.135 - VirusTotal: scan env vars matching `*VIRUSTOTAL*`, `*VT_API*`, `*VT_KEY*`.136137 Record what was found. Do **not** prompt yet if nothing was detected — defer that to Phase 3 handoff.1381392. **Announce connectivity.** Before moving on, surface a single user-facing line summarising what's wired up. Mark each integration `✓` (detected), `—` (missing), or `✗` (probe failed). The Sysdig MCP is always `✓` here — Step 0a already guaranteed it. Example:140141 - `Connectivity: Sysdig MCP ✓ · Jira ✓ · PagerDuty — · VirusTotal —. Will skip binary VT enrichment and PagerDuty handoff.`142143 **Data-egress notice.** Immediately after the connectivity line, when any third-party enrichment is reachable, disclose in one line what indicators will leave the environment and to which service — so the user can veto before any external call:144145 - `Enrichment will send: binary SHA256 hashes → VirusTotal · public (non-private) IPs → ipinfo.io (fallback ip-api.com) · CVE IDs → NVD/CISA KEV/Exploit-DB/GHSA. Private/internal IPs (RFC1918, link-local, loopback, CGNAT) are never sent. No Sysdig event contents, hostnames, or Sysdig credentials leave the environment — the VirusTotal call carries only the hash plus your VirusTotal API key.`146147 Only list the destinations that are actually reachable (`✓`). If the user declines, skip the corresponding enrichment and record it as `skipped: user declined egress`. This is the only narration during Phase 0 — the per-step results above are recorded silently on the case object. Don't prompt beyond the egress veto; just inform.1481493. **Entry-point detection.** Parse the invocation argument:150 - No argument → `interactive` mode.151 - Anything else → `directed` mode, store the value as the event ID.1521534. **Resume check (directed mode only).** If `recent_cases` (from step 0b) has an entry whose `event_id` matches the current trigger, surface a one-line resume summary and ask the user via `AskUserQuestion` how to proceed:154155 > "You investigated `<event_id>` <N> hours ago — case file `<case_file>`, handoff `<destination>:<key>` if any. Continue with the saved case, refresh (re-run Phase 2), or start fresh (clear the prior entry)?"156157 - **Continue** → skip Phase 1 + Phase 2; jump to Phase 3 with the saved case (read the file from disk if it still exists; otherwise treat as `refresh`).158 - **Refresh** → keep the saved entry but re-investigate; merge new findings.159 - **Start fresh** → drop the prior entry from `recent_cases` and re-run.160161 **Staleness default.** If the prior `completed` timestamp is more than 4 hours ago, runtime data has likely shifted — pre-select **Refresh** as the default. Below 4 hours, default to **Continue**.162163 Interactive mode does not pause here — Phase 1 will surface fresh threats and any prior case will only show up in the picker as `(seen <N>h ago)`.164165### Phase 1 — Surface166167**Interactive flow:**1681691. List the top open threats — Threats Engine is exposed via MCP. Call `mcp__secure-mcp-server__list_threats_engine_groups` with `limit=5`, sorted by `lastSignal:desc`, `status=open`.170171 If Threats Engine returns empty (unavailable or quiet in this tenant), fall back to runtime events for the same window: `mcp__secure-mcp-server__list_runtime_events` with last 24h, limit 10.1721732. Present the result as a markdown table:174175 | # | Severity | Rule / aiGeneratedName | Resource | Last seen |176 |---|----------|------------------------|----------|-----------|177178 Ask via `AskUserQuestion`: "Which one do you want to investigate?"1791803. **Incident-scope detection at surface time.** Before diving into the chosen threat, scan the other surfaced groups. Multi-stage attacks frequently span more than one Threats Engine grouping. Treat all groups sharing cluster + ±2h, OR `aws.accountId` + ±2h, OR same image as the **same incident** — one investigation, one case body, one narrative. Record them on `case.incident_threat_groups`.181182 If these conditions hold, the chosen threat is one facet of a larger incident — fold all matching groups into the same case object. Tag them on `case.incident_threat_groups` (id, name, resource, last_seen, why_related). Phase 2's cluster-wide sweep then has a head start — these groups' constituent events should also appear in the sweep, but flagging them upfront lets the report's "Incident scope" section name them by their AI-generated title rather than as anonymous events.183184**Directed flow:**1851861. Resolve the specific threat. The trigger ID may be a threat ID, a group ID, or an event ID — try in that order: `mcp__secure-mcp-server__get_threats_engine_threat` (single-threat detail); on 404 `mcp__secure-mcp-server__get_threats_engine_group`, then `mcp__secure-mcp-server__list_threats_engine_threats_by_group` with the ID; if all 404, fall back to `mcp__secure-mcp-server__get_event_info` with the event ID.187188**Classification — MITRE ATT&CK tactic.** From the rule name, rule source, and event labels, assign one MITRE tactic to the threat. Store it on the case as `case.tactic`. Phase 2 watchlist mapping reads this value. See [`references/mitre-tactics.md`](references/mitre-tactics.md) for the keyword-to-tactic table and the secondary-tactic rule.189190**Process tree (Sysdig MCP).** If the threat has an `event_id` (the Threats Engine returns `securityEvent` references with IDs), call `mcp__secure-mcp-server__get_event_process_tree` with the event ID to retrieve the structured process tree. Store the parsed result (parent → child chain, command lines, sha256 if present) on `case.process_tree`.191192**Process evidence from `aiGeneratedDescription` (always runs — useful even alongside the structured tree).** The description carries natural-language context the structured tree doesn't ("locale repeatedly", "curl --upload to external IP"). Parse it for process names (e.g. `systemd`, `sshd`, `bash`, `curl`, `wget`, `nc`, `nslookup`) and chain hints ("spawned by", "child of"). Store as `case.process_evidence` (list of strings).193194The two are complementary: `case.process_tree` is structured ground truth, `case.process_evidence` is the AI's narrative read of the same chain. The report renders both in "What happened" — the tree as a tree, the evidence as a one-liner.195196Store the threat, classification, secondary tactic (if any), process tree (if available), and process evidence on the working case object.197198### Phase 2 — Investigate (free-form, signal-driven)199200**Goal:** reconstruct the full attack chain starting from the user's pick. Span multiple threat groups, multiple resources, multiple event sources if the signals lead there. The chain is the deliverable — phase boundaries from earlier versions of this skill (e.g. separate enrichment / classifier / synthesis stages) are explicitly *not* prescribed steps anymore.201202Tell the user what you're doing as you go. Examples of good status updates:203204- "Process tree shows Tomcat → bash → xmrig — looks like miner persistence. Looking for related cluster activity."205- "IMDS (Instance Metadata Service) theft on the host. Expanding to CloudTrail in the same AWS account."206- "Found two more threat groups on the same account in the same hour — folding them in as the same campaign."207- "Runtime scan for the workload came back empty — pulling image-level vuln findings instead."208209#### Available signals (chase them when relevant)210211These are the ingredients. The order is yours.212213**Time scoping — read before any event query.** `list_runtime_events`, `count_runtime_events`, and `discover_runtime_event_field_values` accept only `scope_hours` — a window counted *back from now*. There is no absolute start/end, and `filter_expr` has no timestamp field. So a "±2h around the trigger" or "last 7 days" window is **not** something `scope_hours` gives you when the trigger isn't recent: set `scope_hours` wide enough to reach past the trigger (`scope_hours ≈ hours_since_trigger + 2`), then **post-filter the returned events by their timestamp** down to the band you actually want. Treat every "±2h" / "7-day" phrasing below as the *target band after client-side filtering*, not a parameter you can pass directly.214215- **Tenant-wide critical sweep (sanity check)** — once early in Phase 2, call `mcp__secure-mcp-server__list_runtime_events` with `filter_expr = "severity in (0,1,2,3)"` and *no scope filter* across the ±2h window around the trigger. Catches cross-domain signals the cluster/account-filtered queries miss (GitHub `cloudProvider.account.id`, Okta `cloudProvider.tenantId`, anything without K8s labels). Fold in any hit whose image-org, repo name, or actor matches the trigger.216- **Process tree** of the trigger event — `mcp__secure-mcp-server__get_event_process_tree`. Almost always the highest-yield single artifact. Falls back to `aiGeneratedDescription` parsing if the MCP returns empty.217- **Activity audit trail** of the affected workload — the agent-captured record of every command, network connection, and file access around the trigger, independent of whether each tripped a rule. This is what a human analyst pulls in the Secure UI to answer *"what else did this workload do around the detection?"* — dwell time, lateral-movement setup, dropped files, C2 callbacks. Use the MCP tool `list_activity_audit_events` scoped to the **container / pod / workload** (never the bare host) and centred on the trigger time. **Unlike `list_runtime_events`, the activity audit tools take a real absolute window (`from`/`to` as ISO-8601, epoch, or a relative offset like `2h`/`7d`, max 2 weeks)** — so a "±2h around the trigger" band is a direct parameter, *not* the `scope_hours` post-filter workaround. For a suspicious command (dropper, reverse shell, miner), pass its entry id to `trace_activity_audit_command` to reconstruct the child processes, connections, and files it spawned in the same session — the audit-trail counterpart to the process tree. See [`references/activity-audit.md`](references/activity-audit.md) for tools, filter fields, scoping, and the timeline output shape. The trail feeds the "Activity audit timeline" report section and cross-references into the network/binary IOC enrichment.218- **Prior events on the affected resource** — last 7 days, via `mcp__secure-mcp-server__list_runtime_events` with a `filter_expr` matching the workload (`kubernetes.cluster.name + namespace + workload`) or host (`host.hostName`). For K8s workloads, also pull host-level events on the same node — escapes hide there.219- **Cluster-wide activity in a ±2h window** around the trigger. Same MCP tool, three filters in parallel. **The `source` values are tenant/region/backend-specific — confirm the real ones with `mcp__secure-mcp-server__discover_runtime_event_field_values` on the `source` field before filtering, rather than assuming a literal.** Known values at the time of writing: `syscall`, `k8s_audit`, `awscloudtrail` + `awscloudtrail_stateful` (AWS — note there is **no** bare `cloudtrail` value), `azure_platformlogs` + `azure_entra` (Azure), `gcp_auditlog` (GCP). If a `source` filter comes back empty, re-run discover (its `suggested` bucket lists what's actually firing, `other` lists what the tenant knows) and widen to the values it reports.220 - `kubernetes.cluster.name = "<cluster>" and source = "syscall"` (other resources in the cluster)221 - `kubernetes.cluster.name = "<cluster>" and source = "k8s_audit"` (Attach/Exec Pod, Deployment Created, etc.)222 - `kubernetes.cluster.name = "<cluster>" and source in ("awscloudtrail", "awscloudtrail_stateful")` (cluster-tagged cloud events, if any — swap in the Azure/GCP source values from discover when the cluster runs there)223- **Cloud-account-wide activity** when the resource has `aws.accountId` / `azure.subscriptionId` / `gcp.projectId`. Cloud audit-log and agentless-ML events live under the *account* dimension, not the cluster. **This is the difference between catching multi-stage cross-cloud attacks (IMDS credential theft → IAM access-key creation → audit-log tampering → storage exfiltration) and missing them.** Filter by the matching dimension and that cloud's source values (verify with discover first):224 - AWS — `aws.accountId = "<account>" and source in ("awscloudtrail", "awscloudtrail_stateful", "agentless-aws-ml")`225 - Azure — `azure.subscriptionId = "<subscription>" and source in ("azure_platformlogs", "azure_entra")`226 - GCP — `gcp.projectId = "<project>" and source in ("gcp_auditlog")`227- **Other threat groups in this incident** — fold them into the same case — do not investigate separately. Other threat groups already tagged in Phase 1 as part of this incident — pull their constituent events via `mcp__secure-mcp-server__list_threats_engine_threats_by_group` and their affected resources via `mcp__secure-mcp-server__list_threats_engine_resources_by_group` (both keyed by the group ID) to merge into the chain. If new groups appear in the cluster window during Phase 2 investigation, fold them in too. Cross-type is allowed: a CLOUD threat may be the same incident as a K8S_WORKLOAD threat.228- **Sibling resources / posture / RBAC (role-based access control)** via `mcp__secure-mcp-server__run_sysql`. SysQL schema differs between tenants — adjust query shape if rejected. Example queries: `MATCH KubeWorkload AS wl WHERE wl.cluster = '<c>' RETURN wl.namespace, wl.name`, `MATCH Resource VIOLATES Control`, `MATCH KubeServiceAccount HAS KubeRoleBinding HAS KubeClusterRole`.229- **Vulnerability surface** — start with `mcp__secure-mcp-server__query_vulnerability_findings` scoped to the running image, filtered to the notable CVEs:230231 ```json232 {233 "group_by": "cve",234 "filters": [235 { "field": "imageId", "op": "eq", "value": "<running image's config digest>" },236 { "field": "severity", "op": "in", "value": ["critical", "high"] },237 { "field": "hasFix", "op": "eq", "value": true },238 { "field": "hasExploit", "op": "eq", "value": true }239 ]240 }241 ```242243 **Scope by the right identifier.** `imageId` expects the image **config digest** — the `sha256:…` that runtime detection resolves the running container to (the threat detail's image id; equivalently `metadata.imageId` from `get_scan_result` in the scan-result step below). This is a *different* sha256 from the registry **manifest** digest (findingsapi field `imageDigest`, `metadata.digest`) — cross them and the query returns zero rows with no error. If you only have the pullstring, use `imageReference` (e.g. `…/app:1.0.0`) instead, accepting that a mutable tag may span builds.244245 **Filtering.** The tool applies **no** default filters, so set the triage filters explicitly: severity critical/high, `hasFix` true, `hasExploit` true. That narrows to the actionable, weaponized CVEs the MITRE-tactic gate and CTI step below consume. If the result comes back empty, widen by dropping `hasExploit`, then `hasFix`, before concluding the image is clean.246247 `group_by: "cve"` returns one deduplicated row per CVE, ranked by the grouping's default risk order. Read rows from the response's `data` array; `meta.hasMore` + the `cursor` token page further, and `severity` comes back capitalized (`Critical`) in responses even though the filter value is lowercase. For running/in-use package detail — which vulnerable packages are actually loaded on the workload or host — use `mcp__secure-mcp-server__list_runtime_scan_results` (filter by `kubernetes.cluster.name`/`namespace`/`workload` or `host.hostName`, sort `runningVulnTotalBySeverity`) and pass the returned `resultId` to `mcp__secure-mcp-server__get_scan_result` for the full packages + vulnerabilities breakdown.248- **External CTI** for the top 5 critical/high CVEs that pass the MITRE-tactic gate (see `references/correlation-guide.md`): NVD, CISA KEV (Known Exploited Vulnerabilities), Exploit-DB, GHSA (GitHub Security Advisory) via `WebFetch`. Don't fetch CTI for tactic-mismatched CVEs.249- **Sysdig threat-intel feed** via `mcp__secure-mcp-server__fetch_threat_intelligence_feed` — Sysdig-curated CVEs / zero-days / active-attack notes. Cross-reference any IOCs you collect.250- **VirusTotal (VT)** for binary IOCs when a SHA256 surfaces on event fields and a VT key is present (Phase 0 records the env var). When the threat lacks `proc.sha256`, look across other events on the same container — drift detection events typically carry the hash.251- **GeoIP** for network IOCs via `curl https://ipinfo.io/<ip>/json` (fallback `ip-api.com`).252253#### Patterns to recognise (not steps to execute)254255When any of these rule names fire in the data you've pulled, that's the signal to expand. The action column is *what direction to look*, not a fixed query. See [`references/watchlist-patterns.md`](references/watchlist-patterns.md) for the full hit-to-pivot table covering IMDS theft, persistent cloud creds, defense evasion, S3 exfil, privesc, lateral movement, runtime compromise, recon, and prompt-injection.256257#### MITRE-tactic correlation258259Read `references/correlation-guide.md` for the MITRE-tactic gate and 1–5 scoring heuristics. Render only pairs ≥ 4 in the case body. Boosts (cap at 5): KEV +1, VT malicious_count≥5 +1.260261#### Stop conditions262263You're done when:264- The chain has a coherent narrative (initial access → execution → … → impact, or bounded by available data), AND265- You've followed every watchlist hit at least one hop, AND266- You've pulled the activity audit trail for the affected container/pod/workload around the trigger (or recorded why it was unavailable) — the timeline is part of the deliverable, AND267- You've asked yourself *"is there another threat group within the cluster + ±2h / account + ±2h window that belongs to this same incident?"* — and either folded it in or recorded why not.268269If you find yourself making more than ~4 follow-on calls per signal, stop and synthesise. Diminishing returns.270271### Phase 3 — Synthesise & report272273Four steps, in this order:2742751. **Ask the user for the report format** via `AskUserQuestion`: "How do you want the report rendered? `Markdown` (default — plays nice with Jira/PagerDuty handoff) / `HTML` (renders Mermaid diagrams in browser, self-contained) / `Both`."276277 If `AskUserQuestion` is not available, default to `Markdown` and mention HTML as a follow-up option in the summary.2782792. **Write the file(s) to disk.** Base path: `/tmp/sysdig-runtime-investigate-<event_id_short>-<UTC-yyyymmdd-hhmm>`.280 - `Markdown` → write `<base>.md` with Block 1 of `references/reporting-templates.md` verbatim. Always written when chosen — also the body of Jira/PagerDuty handoff downstream.281 - `HTML` → write `<base>.html` using Block 4 wrapper from the same file. The Block 1 markdown content is pasted into the wrapper's `<script id="md">` block. Marked + mermaid CDN scripts render the page in the browser.282 - `Both` → write both files, same base name.2832843. **Print a 2-paragraph summary to the user.** Use Block 0 of `reporting-templates.md` — short, no tables. Cite the file path(s) so they can read it if they want. Then ask `AskUserQuestion`: "Where do you want to report this case?" with options derived from what Phase 0 detected (`Jira`, `PagerDuty`, `Both`, `Just show it`).285286Handoff:287- **Jira** → Build the Block 2 payload from the templates (body = file content). If the project key is unknown, call `mcp__atlassian__getVisibleJiraProjects` and ask the user. **Preview before submitting.** Render a short block — `Project · Issue type · Summary · Severity · Labels · Body length (chars)` — then ask via `AskUserQuestion`: "Submit, edit, or cancel?". Only on **Submit** call `mcp__atlassian__createJiraIssue`. Capture key + URL and surface them with an undo line: *"Undo: open the ticket and click Close."*288- **PagerDuty** → Build the Block 3 payload (`custom_details.case` = file content, truncate to ~30 KB if larger). **Preview before submitting.** Render `Routing key (last 4) · Severity · Dedup key · Summary · Source · custom_details size (KB)` and ask via `AskUserQuestion`: "Submit, edit, or cancel?". Only on **Submit** send via:289290 ```bash291 printf '%s' "$PAYLOAD_JSON" | curl -sS -X POST \292 -H "Content-Type: application/json" \293 --data-binary @- \294 https://events.pagerduty.com/v2/enqueue295 ```296297 Capture the `dedup_key` and incident URL from the response. Surface them with an undo line: *"Undo: resolve the incident with `event_action: resolve` and the same dedup key."*298299- **Both** → run the Jira preview-and-submit first, then the PagerDuty preview-and-submit. Confirm each separately; declining one does not cancel the other.300301- **Just show** → no extra action; the file path was already cited in the summary.302303If the user picks **Edit** at the preview step, ask which fields to change, apply the edits to the payload, and re-show the preview. Loop until the user picks Submit or Cancel.304305After handoff, surface the link/key (if any) to the user — one short line.3063074. **Offer to act on the threat (optional handoff to remediation).** After the reporting handoff is complete (or skipped), ask via `AskUserQuestion`: _"Want me to act on this threat now? `sysdig-runtime-remediate` will propose response actions (kill, isolate, pause, etc.), explain the consequences on the live workload, and execute only what you approve."_ Options: `Yes — invoke sysdig-runtime-remediate` / `No`.308309 - On **Yes** — announce the handoff explicitly per principle C-03 (_"Handing off to `sysdig-runtime-remediate` with `<event_id>`. The case file you just generated will be its starting context."_), then invoke `sysdig-runtime-remediate <event_id>`.310 - On **No** — proceed silently to step 5. The user can always come back later with `/sysdig-runtime-remediate <event_id>` — the case file and shared state will still be there.311312 Skip this step if `sysdig-runtime-remediate` is not registered in the current plugin (back-compat for environments running an older plugin version).3133145. **Persist shared state.** Before exiting, call `mcp__secure-mcp-server__save_skill_state` with `{ "skill_state": "runtime-investigate", "version": <version-from-phase-0>, "data": { ... } }`. Merge the freshly-read state from Phase 0 with the new entries: append this case to `recent_cases` (upsert by `event_id`, cap at 10), update `last_run`, and update `preferred_jira_project` / `preferred_handoff` if the user picked a destination. Skip silently if the Sysdig MCP isn't loaded.315316## Error Handling317318**Status vocabulary.** Every section of the case object uses the house status set: `done`, `pending`, `in_progress`, `failed`, `skipped`. The reason goes in a separate `reason` field so the report and the summary can render either status alone or status + reason.319320```json321{ "vulnerabilities": { "status": "skipped", "reason": "no scan data" } }322{ "cloudtrail": { "status": "skipped", "reason": "no integration in tenant" } }323{ "vt": { "status": "skipped", "reason": "no API key" } }324{ "cti.nvd": { "status": "failed", "reason": "rate-limited" } }325```326327**User-facing error template.** Every error message the user sees follows three lines: **what** failed, **why**, and the **fix** (a copy-pasteable command or concrete next step). Keep the whole thing under four lines. Examples:328329> Can't reach Sysdig — the MCP server isn't registered or authenticated.330> Register it with `claude mcp add --transport http secure-mcp-server https://<region>/mcp/secure`, then run `/mcp` → Sysdig → Authenticate.331> Per-region URLs are in references/mcp-setup.md. Then re-run.332333> No threats matched the last 24h.334> Either the tenant is quiet or the time window is too tight.335> Widen the time window (e.g. 7 days) or pick a specific event ID.336337> Couldn't open the Jira ticket — Atlassian MCP returned 403.338> Your token doesn't have `write:jira-work` for project `RUNTIME`.339> Refresh the token in the Atlassian admin console and retry, or pick another destination.340341Truncation, skipped enrichments, and other partial-success cases must be flagged in the 2-paragraph chat summary too — never let the user discover them only by reading the case file.342343| Situation | Behavior |344|-----------|----------|345| Sysdig MCP preflight returns State 2 or State 3 | Abort in Phase 0a per `auth-preflight.md` — emit its verbatim message and stop. No data calls, no state read, no file writes. |346| An MCP tool call comes back **denied** (the agent refuses the call rather than the server erroring) | Not an auth or reachability failure — the tool isn't on the agent's allowlist, usually a **prefix mismatch**. `allowed-tools` lists the canonical `mcp__secure-mcp-server__*` names (per `mcp-setup.md`), but the server may be registered under a different prefix in this agent (e.g. a plugin-managed name). The bare tool names match 1:1; only the prefix differs. Tell the user: confirm the registered name with `claude mcp list`, then either re-register it as `secure-mcp-server` (the mandated name — see `mcp-setup.md`) or add the actual `mcp__<prefix>__*` pattern to the agent's permitted tools, and re-run. Do not loop retrying the denied call. |347| Trigger ID resolves to neither a threat nor a group (`get_threats_engine_threat`, `get_threats_engine_group`, and `list_threats_engine_threats_by_group` all 404) | Fall back to `mcp__secure-mcp-server__get_event_info` with the same ID. If that 404s too, tell the user the ID wasn't found and offer interactive mode. |348| No qualifying threats / events found in Phase 1 | Tell the user, offer to widen the time window. |349| Vulnerability lookup empty | When `mcp__secure-mcp-server__query_vulnerability_findings` returns an empty `data` array, widen by relaxing filters (drop `hasExploit`, then `hasFix`, then severity) before concluding the image is clean. If still empty, try `mcp__secure-mcp-server__list_runtime_scan_results` (filter by workload/host) then `mcp__secure-mcp-server__get_scan_result`. If still nothing, record `{ "vulnerabilities": { "status": "skipped", "reason": "no scan data" } }` and continue. |350| `WebFetch` to a public CTI source fails or rate-limits | Mark that source `{ "status": "failed", "reason": "<rate-limited \| network \| 4xx>" }` and continue. Don't retry in a loop. |351| VirusTotal API key not detected | Record `{ "vt": { "status": "skipped", "reason": "no API key" } }`. |352| No Jira and no PagerDuty detected | Ask whether the user wants to configure one or just show the file path. |353| Atlassian MCP / PagerDuty curl errors during handoff | Record `{ "handoff": { "status": "failed", "reason": "<error>" } }`, surface the error, print the file path, offer to retry or switch destination. |354| Activity audit trail empty or capture not enabled | `list_activity_audit_events` returns no entries for the scoped container/pod/workload. Not a failure — the agent's command/connection/file capture may be off, or the workload was genuinely quiet. Record `{ "activity_audit": { "status": "skipped", "reason": "no activity audit data for this scope" } }` and flag in the summary that audit context was unavailable. |355| Activity audit window exceeds 2 weeks | `from`/`to` span is capped at 2 weeks server-side. Narrow the window (centre tighter on the trigger) and re-issue. |356| `mcp__secure-mcp-server__list_runtime_events` cursor pagination drops `filter_expr` | Known MCP quirk — paginating with `cursor` returns events from the wrong scope. Don't paginate; widen `scope_hours` and re-issue with the original filter. |357| A list_runtime_events call returns >100 events | Tighten the time window or split the filter. Never silently drop the section — record `{ "status": "done", "truncated": true, "shown": <n>, "total": <m> }`. |358| Cloud audit-log integration (CloudTrail / GCP Audit / Azure Activity) not present in tenant | Cloud-account sweep returns empty (no `source` values for that provider — confirm with discover). Record `{ "cloudtrail": { "status": "skipped", "reason": "no integration in tenant" } }` on the case; flag in the summary that cloud-API signals (storage / IAM / IMDS) were unavailable. |359| SysQL schema rejects a query | Try the alternate relation shapes (`HAS` vs `BINDS_TO`, etc.). If all fail, record `{ "status": "failed", "reason": "schema mismatch" }` and continue. The skill should not block on schema mismatches. |