rmagent-ao — The Agent Observatory
Find and trace every agent on a box, with zero install, from outside.
The OpenLLMetry question — what did this LLM call do? — answered by asking
the box instead of embedding an SDK. SDK-based tracing only sees apps that
opted in. This sees everything, including the shadow Ollama someone installed
last week and the script with a hardcoded key.
The agent-plane sibling of rmagent-so. Same constitution: pull-only,
allowlisted questions, capped answers, holes instead of dumps.
Why agents are unusually observable from outside
A normal web app is a black box at the process boundary. An agent practically
announces itself:
- It calls a small set of known endpoints — api.anthropic.com,
api.openai.com, openrouter, localhost:11434. Network attribution is trivial.
- It spawns tools as child processes — the process tree is the tool-call log.
- It writes transcripts to disk — Claude Code, OpenCode, Aider, Continue,
Goose all persist sessions locally. The data is already there; reading it is
a pull, not an instrument.
- It uses env vars for keys — presence of ANTHROPIC_API_KEY is a cheap,
secret-safe signature.
- Local model servers expose HTTP APIs — Ollama /api/tags + /api/ps,
LM Studio /v1/models, vLLM /v1/models. Already allowlisted-question-shaped.
The questions
| Question |
Returns |
Must NOT return |
agents |
The census — all agent processes, harnesses, frameworks, model servers, endpoints, config paths, packages |
secret values, full transcripts |
agentstate |
One agent's config, models, MCP servers, skills, env-var NAMES (never values), version |
API keys, tokens |
agenttrace |
Recent activity from disk artifacts — sessions, tool calls, files touched, commands run, capped |
full transcript dumps |
agentnet |
Endpoint attribution — which LLM APIs, at what rate/volume, per agent PID |
packet captures |
agentmodels |
Local model servers queried directly — what's installed, what's running |
model weights |
agentdrift |
Baseline + diff — new agents, new endpoints, new models, transcript anomalies |
— |
agentdeep |
ETW burst on one PID (Windows) — 10s of syscall-level tracing, time-boxed, nothing persists |
a persistent agent |
The coverage tiers (stated honestly)
| Tier |
Agents |
Depth |
| 1 — Rich |
CLI agents with disk transcripts (Claude Code, OpenCode, Aider, Continue, Goose, RTerm) |
full sessions, tool calls, commands |
| 2 — Good |
local model servers (Ollama, LM Studio, vLLM, llama.cpp) |
models, running state, metrics via their APIs |
| 3 — Boundary |
any process calling a known LLM endpoint — LangChain/AutoGen/CrewAI apps, custom agents |
process, endpoints, volume, children, files |
| 4 — Opaque |
no disk artifacts, unknown/proxied endpoints, no API |
existence only — recorded as a HOLE, honestly |
The genuine trade vs OpenLLMetry: breadth + zero-install vs inside-depth.
Token counts, prompt text and tool arguments are only available when the agent
writes them to disk (Tier 1) or exposes an API (Tier 2). We get ~80% via
artifacts + boundary + local APIs, and the census finds 100% of what exists.
The scripts
| Script |
Role |
scripts/lib.py |
The engine — door-aware ask(): door=ssh → bash payloads over SSH, door=winrm → PowerShell payloads over pywinrm. Allowlisted, 32 KB cap, holes. |
scripts/questions/linux/*.sh |
Bash payloads for SSH witnesses (Linux/macOS) |
scripts/questions/windows/*.ps1 |
PowerShell payloads for WinRM witnesses |
scripts/stc.py |
Security Trace Context — carries agent= alongside principal= |
scripts/hop_index.py |
Cross-case memory, keyed on (host, agent, session) |
scripts/otel_emit.py |
Optional OTel export — same opt-in as the Flight Recorder |
Inventory
Same shape as the estate. Add the agent skills to a witness's list:
witnesses:
- id: mac
door: ssh
address: localhost
user: olu
skills: [agents, agenttrace, agentnet, agentmodels]
track: [root]
- id: ws1
door: winrm
address: 44.197.31.152
skills: [agents, agentstate, agentnet, agentdrift]
track: [Administrator, SYSTEM]
Laya-assisted agent triage
Every discovered agent (or shadow agent) the census surfaces can be scored by a
fast, typed decision model (Laya, via the use-laya skill) before it reaches the
operator. One matrix lives in decisions/:
agent_triage.json — per census row: is this a shadow or unmanaged agent?
How risky is its configuration? Pull its per-agent questions, flag it, or
record it?
scripts/laya_decide.py agent_triage --state-file agent.json
Policy: confidence below 0.5 is flagged ESCALATE — those rows go to the LLM or
the operator. Verdicts are advisory; the census itself and its capped pulls are
unchanged. The matrix is an allowlist — edit it deliberately, in the light.
Needle tier 0 — offline extraction and drift
Alongside the Laya matrices, every skill in this family can call
needle — a 121M on-device model (Cactus Compute Needle, ~35 MB, no
network, no keys, ~100 MB RAM) installed once on the jump host, never on
the witnesses. It is the free tier of the judgment stack: needle extracts
and compares offline, Laya judges, the LLM reasons.
Two scripts ship in scripts/:
needle_extract.py — typed field extraction from a witness fragment
(count, source_ip, …). Rigid tokens (numbers, IPs) are reliable;
semantic fields are not, so the wrapper validates what it can (:ip,
:int) and blanks anything it cannot trust — never a guess.echo "<attest answer>" | scripts/needle_extract.py user count:int source_ip:ip --json
needle_drift.py — baseline-drift detection via needle's 3072-dim
embeddings: record what an answer normally looks like, score today's
answer against it (similar / drifted / changed, thresholds calibrated
empirically). Separates shape and topic, not field values — value drift
stays the job of the field rules and correlate joins.echo "<routine answer>" | scripts/needle_drift.py record <host>-<question>
echo "<today's answer>" | scripts/needle_drift.py score <host>-<question>
The engine runs in-process on the jump host (the wrapper finds a Python
with cactus-needle — set NEEDLE_PYTHON if it lives elsewhere). No port
is opened, nothing is installed on any witness, and the baselines are
kilobyte holes under ~/.rmagent/needle-drift/, not a lake. On an
air-gapped estate this tier keeps working — and the judgment tier with it: Laya is offline too (pre-seed its HF-cache checkpoint on air-gapped hosts), so no matrix judgment waits for a connected session.
Non-negotiables
- Watch only. No actuation. Never kill an agent process from here.
- Secret values are never read. Env var NAMES only, for the key-presence
signal. No transcript content beyond capped excerpts.
- Capped answers (32 KB). Oversized pulls become holes.
- Your estate only.
- A hole is an answer. Tier 4 agents are recorded as holes, not guessed at.
- WinRM payloads stay under the ~8191-char UTF-16LE budget (same as so).
- Never trust “no agents” until
agents census ran; unknown binary + unknown egress is Tier 4 (blind), not empty.
Relationship to the other skills
| Skill |
Plane |
rmagent-so |
The Security Observatory — identity-led witness questions |
rmagent-fr |
The Flight Recorder — ticket-led tracing |
rmagent-ao |
This skill — the Agent Observatory, agent-led |
rmagent-windows |
The complete Windows skill (so + fr) |
rmagent-redteam |
The drill |
rmagent-actuate |
Phase 1 response |
rmagent-linux |
The Linux/macOS sibling of rmagent-so |
Honest limits
- No inside-view of uninstrumented custom agents (Tier 3/4) — boundary only
- macOS has no eBPF — process/files/network work; deep syscall tracing doesn't
- Windows deep tier is an ETW burst, time-boxed, nothing persistent
- The census is signature-based — a truly novel agent with an unknown endpoint
and no disk artifacts is Tier 4 and shows up as a hole, which is the honest
answer, not a false "no agents found"
Enterprise scope & sizing
- Scale: the census is pull-based and capped (32 KB/agent) — 100 hosts
x 5 probes in minutes; nothing installed on any host; holes (uninstrumented
Tiers 3/4) are reported as holes, never as "no agents".
- Sizing: run the census weekly (or on-demand after change windows);
store per-run JSONL deltas, not full dumps — a year of a 500-host estate is
low MB. Ship deltas to your asset inventory (CMDB) rather than hoarding.
- Cadence: change windows and incident intakes, not a standing loop.
The agent-plane census answers "what agents run here, what do they call,
what changed" — complement EDR (signature plane), don't duplicate it.
- No-Python estates: census targets only need OS-native runtimes
(PowerShell/WinRM on Windows, POSIX sh/SSH on Linux); the jump host that
runs the census engine is yours to place.
1---2name: rmagent-ao3description: >4 The Agent Observatory — find and trace every AI agent, agent harness and5 model server on a box, with zero install, from outside. The agent-plane6 sibling of rmagent-so. A four-probe census (process, network, filesystem,7 packages) discovers all agents including shadow ones the SDK vendors cannot8 see; per-agent questions pull config, models, recent activity from their own9 disk artifacts; local model servers (Ollama, LM Studio, vLLM) are queried10 through their own APIs. Works over SSH (Linux/macOS) and WinRM (Windows).11 Use when you need to know what agents are running on an estate, what they12 are calling, what changed, or to trace one agent's activity — the13 OpenLLMetry question answered by asking the box instead of embedding an SDK.14 Pull-based, capped, holes instead of dumps. Not a lake.15metadata:16 source: https://skillmd.com/skills/drolu/rmagent-ao17---1819# rmagent-ao — The Agent Observatory2021Find and trace every agent on a box, with zero install, from outside.2223**The OpenLLMetry question — what did this LLM call do? — answered by asking24the box instead of embedding an SDK.** SDK-based tracing only sees apps that25opted in. This sees everything, including the shadow Ollama someone installed26last week and the script with a hardcoded key.2728The agent-plane sibling of `rmagent-so`. Same constitution: pull-only,29allowlisted questions, capped answers, holes instead of dumps.3031## Why agents are unusually observable from outside3233A normal web app is a black box at the process boundary. An agent practically34announces itself:3536- It calls a **small set of known endpoints** — api.anthropic.com,37 api.openai.com, openrouter, localhost:11434. Network attribution is trivial.38- It **spawns tools as child processes** — the process tree is the tool-call log.39- It **writes transcripts to disk** — Claude Code, OpenCode, Aider, Continue,40 Goose all persist sessions locally. The data is already there; reading it is41 a pull, not an instrument.42- It uses **env vars for keys** — presence of ANTHROPIC_API_KEY is a cheap,43 secret-safe signature.44- **Local model servers expose HTTP APIs** — Ollama /api/tags + /api/ps,45 LM Studio /v1/models, vLLM /v1/models. Already allowlisted-question-shaped.4647## The questions4849| Question | Returns | Must NOT return |50|---|---|---|51| `agents` | The census — all agent processes, harnesses, frameworks, model servers, endpoints, config paths, packages | secret values, full transcripts |52| `agentstate` | One agent's config, models, MCP servers, skills, env-var NAMES (never values), version | API keys, tokens |53| `agenttrace` | Recent activity from disk artifacts — sessions, tool calls, files touched, commands run, capped | full transcript dumps |54| `agentnet` | Endpoint attribution — which LLM APIs, at what rate/volume, per agent PID | packet captures |55| `agentmodels` | Local model servers queried directly — what's installed, what's running | model weights |56| `agentdrift` | Baseline + diff — new agents, new endpoints, new models, transcript anomalies | — |57| `agentdeep` | ETW burst on one PID (Windows) — 10s of syscall-level tracing, time-boxed, nothing persists | a persistent agent |5859## The coverage tiers (stated honestly)6061| Tier | Agents | Depth |62|---|---|---|63| **1 — Rich** | CLI agents with disk transcripts (Claude Code, OpenCode, Aider, Continue, Goose, RTerm) | full sessions, tool calls, commands |64| **2 — Good** | local model servers (Ollama, LM Studio, vLLM, llama.cpp) | models, running state, metrics via their APIs |65| **3 — Boundary** | any process calling a known LLM endpoint — LangChain/AutoGen/CrewAI apps, custom agents | process, endpoints, volume, children, files |66| **4 — Opaque** | no disk artifacts, unknown/proxied endpoints, no API | existence only — recorded as a HOLE, honestly |6768**The genuine trade vs OpenLLMetry:** breadth + zero-install vs inside-depth.69Token counts, prompt text and tool arguments are only available when the agent70writes them to disk (Tier 1) or exposes an API (Tier 2). We get ~80% via71artifacts + boundary + local APIs, and the census finds 100% of what exists.7273## The scripts7475| Script | Role |76|---|---|77| `scripts/lib.py` | The engine — door-aware `ask()`: `door=ssh` → bash payloads over SSH, `door=winrm` → PowerShell payloads over pywinrm. Allowlisted, 32 KB cap, holes. |78| `scripts/questions/linux/*.sh` | Bash payloads for SSH witnesses (Linux/macOS) |79| `scripts/questions/windows/*.ps1` | PowerShell payloads for WinRM witnesses |80| `scripts/stc.py` | Security Trace Context — carries `agent=` alongside `principal=` |81| `scripts/hop_index.py` | Cross-case memory, keyed on (host, agent, session) |82| `scripts/otel_emit.py` | Optional OTel export — same opt-in as the Flight Recorder |8384## Inventory8586Same shape as the estate. Add the agent skills to a witness's list:8788```yaml89witnesses:90 - id: mac91 door: ssh92 address: localhost93 user: olu94 skills: [agents, agenttrace, agentnet, agentmodels]95 track: [root]96 - id: ws197 door: winrm98 address: 44.197.31.15299 skills: [agents, agentstate, agentnet, agentdrift]100 track: [Administrator, SYSTEM]101```102103## Laya-assisted agent triage104105Every discovered agent (or shadow agent) the census surfaces can be scored by a106fast, typed decision model (Laya, via the `use-laya` skill) before it reaches the107operator. One matrix lives in `decisions/`:108109- `agent_triage.json` — per census row: is this a shadow or unmanaged agent?110 How risky is its configuration? Pull its per-agent questions, flag it, or111 record it?112113```bash114scripts/laya_decide.py agent_triage --state-file agent.json115```116117Policy: confidence below 0.5 is flagged ESCALATE — those rows go to the LLM or118the operator. Verdicts are advisory; the census itself and its capped pulls are119unchanged. The matrix is an allowlist — edit it deliberately, in the light.120121## Needle tier 0 — offline extraction and drift122123Alongside the Laya matrices, every skill in this family can call124**needle** — a 121M on-device model (Cactus Compute Needle, ~35 MB, no125network, no keys, ~100 MB RAM) installed once on the jump host, never on126the witnesses. It is the free tier of the judgment stack: needle extracts127and compares offline, Laya judges, the LLM reasons.128129Two scripts ship in `scripts/`:130131- `needle_extract.py` — typed field extraction from a witness fragment132 (`count`, `source_ip`, …). Rigid tokens (numbers, IPs) are reliable;133 semantic fields are not, so the wrapper validates what it can (`:ip`,134 `:int`) and blanks anything it cannot trust — never a guess.135 ```bash136 echo "<attest answer>" | scripts/needle_extract.py user count:int source_ip:ip --json137 ```138- `needle_drift.py` — baseline-drift detection via needle's 3072-dim139 embeddings: record what an answer normally looks like, score today's140 answer against it (similar / drifted / changed, thresholds calibrated141 empirically). Separates shape and topic, not field values — value drift142 stays the job of the field rules and correlate joins.143 ```bash144 echo "<routine answer>" | scripts/needle_drift.py record <host>-<question>145 echo "<today's answer>" | scripts/needle_drift.py score <host>-<question>146 ```147148The engine runs in-process on the jump host (the wrapper finds a Python149with `cactus-needle` — set `NEEDLE_PYTHON` if it lives elsewhere). No port150is opened, nothing is installed on any witness, and the baselines are151kilobyte holes under `~/.rmagent/needle-drift/`, not a lake. On an152air-gapped estate this tier keeps working — and the judgment tier with it: Laya is offline too (pre-seed its HF-cache checkpoint on air-gapped hosts), so no matrix judgment waits for a connected session.153154## Non-negotiables155156- **Watch only.** No actuation. Never kill an agent process from here.157- **Secret values are never read.** Env var NAMES only, for the key-presence158 signal. No transcript content beyond capped excerpts.159- **Capped answers** (32 KB). Oversized pulls become holes.160- **Your estate only.**161- **A hole is an answer.** Tier 4 agents are recorded as holes, not guessed at.162- **WinRM payloads stay under the ~8191-char UTF-16LE budget** (same as so).163- **Never trust “no agents”** until `agents` census ran; unknown binary + unknown egress is Tier 4 (blind), not empty.164165## Relationship to the other skills166167| Skill | Plane |168|---|---|169| `rmagent-so` | The Security Observatory — identity-led witness questions |170| `rmagent-fr` | The Flight Recorder — ticket-led tracing |171| `rmagent-ao` | **This skill — the Agent Observatory, agent-led** |172| `rmagent-windows` | The complete Windows skill (so + fr) |173| `rmagent-redteam` | The drill |174| `rmagent-actuate` | Phase 1 response |175| `rmagent-linux` | The Linux/macOS sibling of rmagent-so |176177## Honest limits1781791. No inside-view of uninstrumented custom agents (Tier 3/4) — boundary only1802. macOS has no eBPF — process/files/network work; deep syscall tracing doesn't1813. Windows deep tier is an ETW burst, time-boxed, nothing persistent1824. The census is signature-based — a truly novel agent with an unknown endpoint183 and no disk artifacts is Tier 4 and shows up as a hole, which is the honest184 answer, not a false "no agents found"185186187---188189## Enterprise scope & sizing190191- **Scale:** the census is pull-based and capped (32 KB/agent) — 100 hosts192 x 5 probes in minutes; nothing installed on any host; holes (uninstrumented193 Tiers 3/4) are reported as holes, never as "no agents".194- **Sizing:** run the census weekly (or on-demand after change windows);195 store per-run JSONL deltas, not full dumps — a year of a 500-host estate is196 low MB. Ship deltas to your asset inventory (CMDB) rather than hoarding.197- **Cadence:** change windows and incident intakes, not a standing loop.198 The agent-plane census answers "what agents run here, what do they call,199 what changed" — complement EDR (signature plane), don't duplicate it.200- **No-Python estates:** census targets only need OS-native runtimes201 (PowerShell/WinRM on Windows, POSIX sh/SSH on Linux); the jump host that202 runs the census engine is yours to place.