rmagent-so — The Security Observatory
The witness-question half of the RMAgent security observatory: ask a small
set of allowlisted named questions, track the identities that matter, and
never copy the log home.
This is the Security Observatory engine, split out of rmagent-windows so the
question half can be reasoned about (and loaded) on its own. rmagent-windows
remains the complete, runnable skill; rmagent-fr is the Flight Recorder
(tracing) half. The three skills share one constitution.
Non-negotiables
- Watch only. No
actuate. No systemctl stop, no usermod, no firewall edits.
- Allowlisted questions only — never an arbitrary shell.
- Capped answers (32 KB) — oversized pulls become holes, never a lake.
- Your estate only. No box you do not administer.
- No tight retry on a silent host.
- Credentials never in the inventory file — env or
~/.rmagent/creds.json.
- Never trust a "no findings" result until you have confirmed the witness can see.
The questions
| Question |
Returns |
Must NOT return |
attest |
alive, last boot, admin failed/ok logons, local admin count, SYSTEM conns, raw_4624_24h + blind_check + blind_count + oldest_security_event |
a full Security log dump |
sketch |
admin failed in window, new local admins 24h, running priv services, new services/tasks |
raw event lists |
edges |
tracked logons (src IP, LogonId, auth pkg) + explicit-cred uses (4648) + special-priv grants (4672) + outbound conns, capped |
the whole connection table |
explain |
identity/svc/task changes + 4648/4672 + WMI subs + audit-cleared (1102) + LOLBin spawns w/ cmdline, capped |
the whole ring/tenant export |
netedges |
Sysmon ring: conns + DNS + LSASS access (T1003) + thread injection (T1055) + file creates + registry sets |
the full netflow |
pslogs |
PowerShell script blocks (4104) — the ACTUAL CODE being executed, decompiled |
the whole PowerShell log |
kernring |
10-second burst capture of process events from the Sysmon ring |
a persistent agent |
attackmap |
13 registry persistence locations, ATT&CK-tagged, FP-allowlisted |
the whole filesystem |
flowstats |
per-adapter byte totals + top destinations (the T1041 volume baseline) |
the full packet capture |
deepwindow |
a short-lived ETW kernel trace, captured at full fidelity, stopped, read back |
a persistent agent |
canary |
any auth attempt against a decoy identity (4624/4625/4740) + the source IPs |
anything about real accounts |
canary — the patient-zero tripwire (Rev 15)
A canary is a decoy identity that exists only to be touched. Any
authentication attempt against it is critical by definition — there is no
legitimate reason to log on as honeyadmin. This turns patient-zero
detection from a graph walk into a tripwire: near-zero false positive, no
correlation needed.
- Declare canaries in the inventory:
canaries: [honeyadmin, svcbackup2]
- With none declared, the payload falls back to decoy-name heuristics
(
canary, honey, decoy, tripwire, …) so an estate that planted
decoys without updating the inventory still gets coverage
- A hit surfaces as a critical
canary_tripped finding in both drift
and correlate, carrying the source IPs — the shortlist for
actuate.py block_ip
The scripts
| Script |
Role |
scripts/stc.py |
Security Trace Context — the W3C traceparent analog. Context propagates across hosts; event data never does |
scripts/lib.py |
The engine — allowlisted ask(), signal-aware cap, holes, pywinrm transport, scrt credential fallback, attackmap FP allowlist. Rev 17: unparseable answers are holes (not clean), credentials never touch os.environ, and the silent-host cooldown (mark_silent/cooldown_left) enforces "no tight retry on a silent host" in code |
scripts/hunt.py |
The walk — edges → explain → pslogs where smoke; smoke → Telegram. Runs correlate at the end. Rev 17: skips witnesses in cooldown and records the skip as a hole |
scripts/correlate.py |
Cross-witness join — cross-host-account, lateral-hop (critical), explicit-cred-to-peer, shared-logonid (critical), canary_tripped. Adds triage ranking + recommended actuate actions. Rev 17: the shared-logonid join is the (LogonId, user) PAIR within 10 minutes (per-boot coincidences downgrade to info) and the peer match is an exact token, not a substring |
scripts/patient_zero.py |
The backward graph walk with honest termination — origin vs retention-boundary vs blind-witness vs no-signal vs cycle |
scripts/drift.py |
Baseline + diff — new_admins (critical), sysmon_change (critical), witness_blind (critical), new_persistence, canary_tripped (critical) |
scripts/case.py |
Open / list / close a one-page case. --ticket PAY-4419 threads a business id through everything. Rev 17: prune --days N sheds old cases' raw answers (the story is kept) — a lake by accretion is still a lake |
scripts/notify.py |
Telegram helper (token+chat from secrets store). |
scripts/test_enterprise.py |
Pure-logic test suite for all of the above (57 assertions). |
scripts/test_budget.py |
Enforces the WinRM ~8191-char budget on every payload. |
scripts/questions/windows/*.ps1 |
The payloads above. Compact — preamble+payload must encode under WinRM's ~8191-char budget (×2.7 for UTF-16LE base64). Rev 17: every event query carries -MaxEvents so the DEVICE sheds first (a flood cannot balloon the scan past the timeout), and tracked-name matching is exact on the bare name |
The signal-aware cap (Rev 15)
The flat 32 KB cap was an evasion surface: a noisy host (or an attacker
flooding events) pushed the signal past the window and the whole answer
became a hole — the loudest box got ignored.
Now an over-budget answer is triaged, not dropped: low-signal rows are
shed first, and rows carrying critical event IDs (4648, 4672, 5861, 1102,
4104, 4698, 7045, 4732, 4688) always survive the trim. The cap is never
raised — we only choose what survives it. Still no lake.
An answer that was trimmed carries capped: true and a cap_note, so the
operator knows the window was narrowed rather than being silently deceived.
Security Trace Context (STC) — context propagates, data does not
The W3C traceparent analog, but the unit is a principal's walk across an
estate, not a request across services:
stc: case=CASE-20260825-143022; principal=Administrator;
window=2h; origin=jh1; depth=2; ticket=PAY-4419;
apptrace=4bf92f3577b34da6a3ce929d0e0e4736
- Context travels; event data never does. When a hunt moves from WS1 to
WS2, WS2 receives "I'm looking for Administrator, case X, depth 2" — not
WS1's logs. Each host answers from its own local tools.
depth is the distributed circuit breaker. The walk budget (≤ 8) applies
across the WHOLE trace, not per-host — this is what stops a distributed
hunt becoming a worm.
ticket is the business join (a payment id, incident number) — the
Flight Recorder link.
apptrace (v2) is the application trace id — an OTel trace_id or W3C
traceparent observed on a witness. It lets ONE case serve both lenses:
"who walked?" (principal) and "which request was slow?" (app trace).
Identity-led and request-led correlation on the same tape.
Start a hunt joined to an application request:
python3 hunt.py --inventory estate.yaml --since 2h \
--ticket PAY-4419 --app-trace-id 4bf92f3577b34da6a3ce929d0e0e4736
Every emitted OTel span then carries rmagent.app_trace_id, so in
Grafana/Jaeger the security walk and the application request appear as
correlatable traces — one waterfall for "the request was slow", one for
"and here is who was walking the estate at that moment".
Injection-hardened: the STC is a delimiter format, so a crafted value
like X; depth=9; principal=root could rewrite the walk budget. Values
containing ; or = are rejected at construction AND at decode, and
duplicate keys are rejected at decode. test_stc_v2.py asserts all of it.
Triage (Rev 15)
With 50 findings across 10 hosts, an operator needs to know what to actuate
FIRST. Every correlate finding now carries a triage_rank, a triage_why,
and recommended_actions (drawn only from the actuate allowlist):
| rank |
kind |
why |
actuate |
| 0 |
canary_tripped |
near-zero FP, patient-zero candidate |
block_ip, disable_user, kill_process |
| 1 |
shared-logonid |
stolen credential in active use |
disable_user, block_ip |
| 2 |
lateral-hop |
active movement |
block_ip, kill_process |
| 3 |
new_admins |
privilege gained since baseline |
remove_admin, disable_user |
| 4 |
new_tracked_proc |
new process as a tracked principal |
kill_process, quarantine_file |
| 5 |
witness_blind |
every other answer is suspect |
(policy fix, not actuate) |
| 6 |
explicit-cred-to-peer |
explicit creds against a peer |
disable_user |
| 7 |
cross-host-account |
could be legitimate admin |
disable_user |
| 8 |
sysmon_change |
the tripwire itself moved |
(investigate) |
| 9 |
new_persistence |
persistence grew |
delete_task, stop_service, disable_wmi_sub |
The blind check (standing rule)
Found live on WS2: the Logon audit policy was Failure-only, so edges returned
zero logons while an Administrator session was connected. Every "clean"
report from that box was a silent false negative.
Every attest now carries raw_4624_24h (unfiltered 4624 count), blind_check
(per-subcategory ok/BLIND/unknown for the sources the questions depend on), and
blind_count (must be 0). drift treats a growing blind_count as critical.
Rev 17 hardens the check itself:
- Locale-invariant.
auditpol is parsed by SUBCATEGORY GUID (the {0CCE...}
column in the CSV), not display name — a non-English Windows no longer reports
every entry as unknown, which used to make blind_count=0 and the box look
sighted.
- Two new sources the questions depend on:
Process CmdLine (4688 command-line
inclusion — without it lolbin_spawns.c is null and the LOLBin detection is
decorative) and ScriptBlock Logging (4104 — without it pslogs is ALWAYS
empty). These are the two an attacker is most likely to switch off.
Standing rule: never trust a "no findings" result until you have confirmed
the witness can see. An empty answer is not a clean answer — and (Rev 17)
neither is an unparseable one: a payload that fails to emit JSON is a hole,
not ok with a raw blob.
Relationship to the other skills
| Skill |
Half |
rmagent-windows |
The complete skill — both halves, fully runnable |
rmagent-fr |
The Flight Recorder (tracing) half |
rmagent-so |
This skill — the witness-question (Security Observatory) half |
rmagent-redteam |
The drill — stages artifacts, scores detection |
rmagent-actuate |
Phase 1 response — named, journaled, reversible |
rmagent-linux |
The Linux/macOS sibling of this skill |
1---2name: rmagent-so3description: The Security Observatory — the witness-question half of the RMAgent security observatory. Pull-based, allowlisted named questions over WinRM :5985 tracking Administrator and SYSTEM: attest (with blind_check — can this witness actually see?), sketch, edges, explain, netedges, pslogs, kernring, attackmap, flowstats, deepwindow. Cross-witness correlation joins the answers (cross-host-account, lateral-hop, shared-logonid); baseline drift flags new admins and witness blindness as critical. Use for identity-led compromise, lateral movement, living-off-the-land, silent hosts, and honest root-cause on Windows boxes you administer. Watch-only, capped at 32 KB, holes instead of dumps — no lake, no agent install.4---56# rmagent-so — The Security Observatory78The witness-question half of the RMAgent security observatory: **ask a small9set of allowlisted named questions, track the identities that matter, and10never copy the log home.**1112This is the Security Observatory engine, split out of `rmagent-windows` so the13question half can be reasoned about (and loaded) on its own. `rmagent-windows`14remains the complete, runnable skill; `rmagent-fr` is the Flight Recorder15(tracing) half. The three skills share one constitution.1617## Non-negotiables1819- **Watch only.** No `actuate`. No `systemctl stop`, no `usermod`, no firewall edits.20- **Allowlisted questions only** — never an arbitrary shell.21- **Capped answers** (32 KB) — oversized pulls become holes, never a lake.22- **Your estate only.** No box you do not administer.23- **No tight retry** on a silent host.24- **Credentials never in the inventory file** — env or `~/.rmagent/creds.json`.25- **Never trust a "no findings" result until you have confirmed the witness can see.**2627## The questions2829| Question | Returns | Must NOT return |30|---|---|---|31| `attest` | alive, last boot, admin failed/ok logons, local admin count, SYSTEM conns, **raw_4624_24h + blind_check + blind_count + oldest_security_event** | a full Security log dump |32| `sketch` | admin failed in window, new local admins 24h, running priv services, new services/tasks | raw event lists |33| `edges` | tracked logons (src IP, LogonId, auth pkg) + explicit-cred uses (4648) + special-priv grants (4672) + outbound conns, capped | the whole connection table |34| `explain` | identity/svc/task changes + 4648/4672 + WMI subs + audit-cleared (1102) + LOLBin spawns w/ cmdline, capped | the whole ring/tenant export |35| `netedges` | Sysmon ring: conns + DNS + LSASS access (T1003) + thread injection (T1055) + file creates + registry sets | the full netflow |36| `pslogs` | PowerShell script blocks (4104) — the ACTUAL CODE being executed, decompiled | the whole PowerShell log |37| `kernring` | 10-second burst capture of process events from the Sysmon ring | a persistent agent |38| `attackmap` | 13 registry persistence locations, ATT&CK-tagged, FP-allowlisted | the whole filesystem |39| `flowstats` | per-adapter byte totals + top destinations (the T1041 volume baseline) | the full packet capture |40| `deepwindow` | a short-lived ETW kernel trace, captured at full fidelity, stopped, read back | a persistent agent |41| `canary` | any auth attempt against a decoy identity (4624/4625/4740) + the source IPs | anything about real accounts |4243### `canary` — the patient-zero tripwire (Rev 15)4445A canary is a decoy identity that exists **only to be touched**. Any46authentication attempt against it is critical by definition — there is no47legitimate reason to log on as `honeyadmin`. This turns patient-zero48detection from a graph walk into a tripwire: near-zero false positive, no49correlation needed.5051- Declare canaries in the inventory: `canaries: [honeyadmin, svcbackup2]`52- With none declared, the payload falls back to decoy-name heuristics53 (`canary`, `honey`, `decoy`, `tripwire`, …) so an estate that planted54 decoys without updating the inventory still gets coverage55- A hit surfaces as a **critical** `canary_tripped` finding in both `drift`56 and `correlate`, carrying the source IPs — the shortlist for57 `actuate.py block_ip`5859## The scripts6061| Script | Role |62|---|---|63| `scripts/stc.py` | Security Trace Context — the W3C `traceparent` analog. Context propagates across hosts; event data never does |64| `scripts/lib.py` | The engine — allowlisted `ask()`, **signal-aware cap**, holes, pywinrm transport, scrt credential fallback, attackmap FP allowlist. Rev 17: unparseable answers are holes (not clean), credentials never touch `os.environ`, and the **silent-host cooldown** (`mark_silent`/`cooldown_left`) enforces "no tight retry on a silent host" in code |65| `scripts/hunt.py` | The walk — edges → explain → pslogs where smoke; smoke → Telegram. Runs correlate at the end. Rev 17: skips witnesses in cooldown and records the skip as a hole |66| `scripts/correlate.py` | Cross-witness join — cross-host-account, lateral-hop (critical), explicit-cred-to-peer, shared-logonid (critical), canary_tripped. **Adds triage ranking + recommended actuate actions.** Rev 17: the shared-logonid join is the (LogonId, user) PAIR within 10 minutes (per-boot coincidences downgrade to info) and the peer match is an exact token, not a substring |67| `scripts/patient_zero.py` | The backward graph walk with **honest termination** — origin vs retention-boundary vs blind-witness vs no-signal vs cycle |68| `scripts/drift.py` | Baseline + diff — new_admins (critical), sysmon_change (critical), witness_blind (critical), new_persistence, **canary_tripped (critical)** |69| `scripts/case.py` | Open / list / close a one-page case. `--ticket PAY-4419` threads a business id through everything. Rev 17: `prune --days N` sheds old cases' raw answers (the story is kept) — a lake by accretion is still a lake |70| `scripts/notify.py` | Telegram helper (token+chat from secrets store). |71| `scripts/test_enterprise.py` | Pure-logic test suite for all of the above (57 assertions). |72| `scripts/test_budget.py` | Enforces the WinRM ~8191-char budget on every payload. |73| `scripts/questions/windows/*.ps1` | The payloads above. Compact — preamble+payload must encode under WinRM's ~8191-char budget (×2.7 for UTF-16LE base64). Rev 17: every event query carries `-MaxEvents` so the DEVICE sheds first (a flood cannot balloon the scan past the timeout), and tracked-name matching is exact on the bare name |7475### The signal-aware cap (Rev 15)7677The flat 32 KB cap was an **evasion surface**: a noisy host (or an attacker78flooding events) pushed the signal past the window and the whole answer79became a hole — the loudest box got ignored.8081Now an over-budget answer is **triaged, not dropped**: low-signal rows are82shed first, and rows carrying critical event IDs (4648, 4672, 5861, 1102,834104, 4698, 7045, 4732, 4688) always survive the trim. The cap is never84raised — we only choose *what* survives it. Still no lake.8586An answer that was trimmed carries `capped: true` and a `cap_note`, so the87operator knows the window was narrowed rather than being silently deceived.8889### Security Trace Context (STC) — context propagates, data does not9091The W3C `traceparent` analog, but the unit is a **principal's walk across an92estate**, not a request across services:9394```95stc: case=CASE-20260825-143022; principal=Administrator;96 window=2h; origin=jh1; depth=2; ticket=PAY-4419;97 apptrace=4bf92f3577b34da6a3ce929d0e0e473698```99100- **Context travels; event data never does.** When a hunt moves from WS1 to101 WS2, WS2 receives "I'm looking for Administrator, case X, depth 2" — not102 WS1's logs. Each host answers from its own local tools.103- **`depth` is the distributed circuit breaker.** The walk budget (≤ 8) applies104 across the WHOLE trace, not per-host — this is what stops a distributed105 hunt becoming a worm.106- **`ticket`** is the business join (a payment id, incident number) — the107 Flight Recorder link.108- **`apptrace` (v2)** is the application trace id — an OTel `trace_id` or W3C109 `traceparent` observed on a witness. It lets ONE case serve both lenses:110 *"who walked?"* (principal) **and** *"which request was slow?"* (app trace).111 Identity-led and request-led correlation on the same tape.112113Start a hunt joined to an application request:114115```bash116python3 hunt.py --inventory estate.yaml --since 2h \117 --ticket PAY-4419 --app-trace-id 4bf92f3577b34da6a3ce929d0e0e4736118```119120Every emitted OTel span then carries `rmagent.app_trace_id`, so in121Grafana/Jaeger the security walk and the application request appear as122correlatable traces — one waterfall for "the request was slow", one for123"and here is who was walking the estate at that moment".124125**Injection-hardened:** the STC is a delimiter format, so a crafted value126like `X; depth=9; principal=root` could rewrite the walk budget. Values127containing `;` or `=` are rejected at construction AND at decode, and128duplicate keys are rejected at decode. `test_stc_v2.py` asserts all of it.129130### Triage (Rev 15)131132With 50 findings across 10 hosts, an operator needs to know what to actuate133FIRST. Every correlate finding now carries a `triage_rank`, a `triage_why`,134and `recommended_actions` (drawn only from the actuate allowlist):135136| rank | kind | why | actuate |137|---|---|---|---|138| 0 | `canary_tripped` | near-zero FP, patient-zero candidate | block_ip, disable_user, kill_process |139| 1 | `shared-logonid` | stolen credential in active use | disable_user, block_ip |140| 2 | `lateral-hop` | active movement | block_ip, kill_process |141| 3 | `new_admins` | privilege gained since baseline | remove_admin, disable_user |142| 4 | `new_tracked_proc` | new process as a tracked principal | kill_process, quarantine_file |143| 5 | `witness_blind` | every other answer is suspect | *(policy fix, not actuate)* |144| 6 | `explicit-cred-to-peer` | explicit creds against a peer | disable_user |145| 7 | `cross-host-account` | could be legitimate admin | disable_user |146| 8 | `sysmon_change` | the tripwire itself moved | *(investigate)* |147| 9 | `new_persistence` | persistence grew | delete_task, stop_service, disable_wmi_sub |148149## The blind check (standing rule)150151Found live on WS2: the Logon audit policy was Failure-only, so `edges` returned152**zero logons while an Administrator session was connected**. Every "clean"153report from that box was a silent false negative.154155Every `attest` now carries `raw_4624_24h` (unfiltered 4624 count), `blind_check`156(per-subcategory ok/BLIND/unknown for the sources the questions depend on), and157`blind_count` (must be 0). `drift` treats a growing `blind_count` as **critical**.158159Rev 17 hardens the check itself:160- **Locale-invariant.** `auditpol` is parsed by SUBCATEGORY GUID (the `{0CCE...}`161 column in the CSV), not display name — a non-English Windows no longer reports162 every entry as `unknown`, which used to make `blind_count=0` and the box look163 sighted.164- **Two new sources the questions depend on:** `Process CmdLine` (4688 command-line165 inclusion — without it `lolbin_spawns.c` is null and the LOLBin detection is166 decorative) and `ScriptBlock Logging` (4104 — without it `pslogs` is ALWAYS167 empty). These are the two an attacker is most likely to switch off.168169**Standing rule: never trust a "no findings" result until you have confirmed170the witness can see.** An empty answer is not a clean answer — and (Rev 17)171neither is an unparseable one: a payload that fails to emit JSON is a hole,172not `ok` with a raw blob.173174## Relationship to the other skills175176| Skill | Half |177|---|---|178| `rmagent-windows` | The complete skill — both halves, fully runnable |179| `rmagent-fr` | The Flight Recorder (tracing) half |180| `rmagent-so` | This skill — the witness-question (Security Observatory) half |181| `rmagent-redteam` | The drill — stages artifacts, scores detection |182| `rmagent-actuate` | Phase 1 response — named, journaled, reversible |183| `rmagent-linux` | The Linux/macOS sibling of this skill |