audit-activation
Reports whether OrchestKit's specialized sub-agents are actually being activated, from real spawn telemetry — not vibes. Scope is agent activation, not skill quality.
It answers: "Do my specialized agents get spawned, or does the model default to generic Explore/general-purpose?" Read-only — it never edits agents.
Why this exists: a 2026-06 audit found only ~14% of agent spawns hit the 36-agent catalog vs ~74% generic, with 17/36 agents dormant — and that agents fire ~1:1 with how often a high-traffic skill names them via subagent_type=. Description rewrites ("use proactively") were A/B-tested and gave Δ0, so this skill measures wiring + usage, not description prose. See docs/feat--activation-audit/.
Quick Reference
| Category |
File |
Impact |
When to Use |
| Activation Checks |
rules/activation-checks.md |
HIGH |
What to compute per agent |
| Classification |
rules/activation-status.md |
HIGH |
fires / mis-triggered / niche / dead buckets |
| Output Format |
references/output-format.md |
MEDIUM |
Report layout + the spawn-split summary |
CRITICAL: Task Management is MANDATORY (CC 2.1.16)
TaskCreate(subject="Audit activation: agent spawn analysis",
description="Computing generic-vs-specialist split + dormancy from spawn telemetry",
activeForm="Auditing agent activation")
TaskCreate(subject="Read spawn telemetry", activeForm="Reading subagent-spawns.jsonl")
TaskCreate(subject="Compute split + observed zero starts", activeForm="Computing activation events and coverage")
TaskCreate(subject="Classify & render", activeForm="Classifying agents and rendering report")
TaskUpdate(taskId="3", addBlockedBy=["2"])
TaskUpdate(taskId="4", addBlockedBy=["3"])
Workflow
Run the script FIRST — every audit starts by running (or, when execution is impossible, explicitly referencing) the deterministic collector:
bash "${CLAUDE_SKILL_DIR}/scripts/run-activation-audit.sh" \
--telemetry-root /path/to/consumer-project \
--stop-feed "$HOME/.claude/analytics/agent-usage.jsonl" \
--days 30 --json
It resolves the catalog beside its source or installed script, while every consumer telemetry root is explicit. It reports attempts, starts, and stop-feed completions separately. Never eyeball JSONL by hand when the script exists.
Inventory — let the script resolve its own source or installed agents/ catalog. Do not point it at a consumer project's agent directory.
Read telemetry — .claude/logs/subagent-spawns.jsonl carries pretool intent and start events. Triangulate it with the restored global ~/.claude/analytics/agent-usage.jsonl stop feed when available. The streams have no stable common event ID, so never add them together or infer unique spawns.
Compute — all checks from Read("rules/activation-checks.md"); the Report Contract below lists the mandatory ones.
Classify — bucket every agent using the Four Buckets below (full procedure: Read("rules/activation-status.md")).
Render — output per Read("references/output-format.md"), satisfying the Report Contract.
Report Contract (every audit MUST include all six)
- Data-source line (first line of the report) — name
scripts/run-activation-audit.sh, every explicit consumer .claude/logs/subagent-spawns.jsonl root, the UTC window, and the spawn coverage state (observed, missing, empty, or partial). State whether ~/.claude/analytics/agent-usage.jsonl was supplied as a stop feed. A report that presents numbers without these coverage facts is invalid.
- Spawn split — generic (Explore/general-purpose/Plan) vs ork-catalog vs other-plugin, as counts AND percentages (e.g. "412 generic (74%) / 78 ork-catalog (14%) / 67 other (12%)"). Never percentages alone.
- Concentration — the top-5 agents' share of all catalog spawns as a percentage (e.g. "top-5 = 81% of catalog spawns"), plus the top-5 list with fire counts.
- Observed-zero-start list — only when spawn coverage is
observed, enumerate every catalog agent with zero start events in the supplied roots, one per line with its reference count and bucket. This is not an estate-wide or lifetime dormancy claim. Do not render the list when coverage is missing, empty, or partial.
- Window caveat — state that observed-zero-start names are absent only from the supplied roots and UTC window. They are a routing signal, not proof of zero estate-wide or lifetime use.
- Four-bucket classification table (below) + fix recommendations — wiring changes only (see Hard Rules).
Classification: the Four Buckets (rules/activation-status.md)
Bucket by reference counts, never by description quality. For each observed-zero-start agent, count its real spawn references: grep -rc "subagent_type=ork:<name>" src/skills/ (plus agent:/team-map mentions in src/agents/). Show the evidence: cite that grep command in the report and put each agent's ref-count (with an example source file, e.g. src/skills/cover/SKILL.md) in the classification table. The telemetry observation proves only starts in supplied roots; the grep proves wiring. Render classification only when coverage is observed.
| Bucket |
Condition |
Action |
| ACTIVE |
fired ≥1× in the telemetry window |
none — it works |
| MIS-TRIGGERED |
observed zero starts, has references but no real subagent_type= spawn from a busy skill |
wire a spawn from a high-traffic skill |
| NICHE |
observed zero starts, rare-by-nature domain (design, multimodal, perf, IaC, security-LLM) AND wired somewhere |
keep - low use expected |
| DEAD |
observed zero starts AND zero references in ANY skill or agent |
prune candidate |
DEAD requires zero references anywhere — an agent with even one reference is never DEAD; it is mis-triggered or niche. Only DEAD agents are prune candidates.
Hard Rules
- Never recommend rewriting agent descriptions. A/B-tested: description rewrites gave Δ0 routing benefit. Dormancy is a skill-wiring problem — the fix is always adding a
subagent_type=ork:<name> spawn from a high-traffic skill (e.g. implement, cover, review-pr), never prose changes.
- Classify by reference count, not description quality. The grep above is the evidence; "the description is vague" is not.
- State the telemetry window, supplied roots, and coverage state in EVERY answer — including classification-only or prune-decision answers. Observed-zero-start is never an estate-wide dormancy claim.
- Keep event semantics separate. Pretool is attempted intent,
source:start is started activation, and agent-usage.jsonl is a stop/completion observation. Cite all supplied sources and never sum them into a unique-spawn claim.
- Answer with the report itself, never with only a description of the process. Even when you cannot execute the script in the current context, render the full contract-format report (state which numbers are from the latest available run vs illustrative).
- Read-only: this skill never edits agents.
Example report shape (illustrative numbers — imitate the FORM exactly)
Data: scripts/run-activation-audit.sh over /consumer/.claude/logs/subagent-spawns.jsonl (window: 2026-05-23T00:00:00Z to 2026-07-14T00:00:00Z; coverage: observed)
Stop feed: ~/.claude/analytics/agent-usage.jsonl supplied separately; it is not joined to starts.
## Spawn split
Total 560 spawns: 412 generic (74%) / 78 ork-catalog (14%) / 70 other-plugin (12%)
## Concentration
Top-5 = 63/78 catalog spawns (81%): web-research-analyst 26, code-quality-reviewer 21,
test-generator 8, debug-investigator 5, backend-system-architect 3
## Observed zero-start agents (6 of 20 in the supplied roots, refs via grep -rc "subagent_type=ork:<name>" src/skills/)
- emulate-engineer (3 refs) -> MIS-TRIGGERED
- expect-agent (3 refs) -> MIS-TRIGGERED
- design-system-architect (2 refs) -> NICHE (rare domain, wired)
- multimodal-specialist (1 ref) -> NICHE (rare domain, wired)
- infrastructure-architect (2 refs) -> NICHE (rare domain, wired)
- event-driven-architect (1 ref) -> MIS-TRIGGERED
Caveat: observed zero starts are absent only from the supplied telemetry roots and
window. They are not proof of zero estate-wide or lifetime use.
## Buckets (rules/activation-status.md)
ACTIVE (14) | MIS-TRIGGERED (3) | NICHE (3) | DEAD (0 — none: every agent has >=1 reference)
## Fixes (wiring only — never description rewrites)
- emulate-engineer: add subagent_type=ork:emulate-engineer spawn from cover (src/skills/cover/SKILL.md)
- expect-agent: add subagent_type=ork:expect-agent spawn from review-pr
Chain
After this audit, run the deeper experiment if you suspect descriptions: docs/feat--activation-audit/agent-routing-experiment.mjs (isolated A/B selection harness).
Related Skills
telemetry-inspect — validates the telemetry data-plane this skill reads fromanalytics — raw usage queries across projectsdoctor — broader plugin health (manifests, hooks, memory budget)
Eval note: this skill is direct-only (disable-model-invocation: true), so quality evals run in TIER-1 unit mode (--force-skill) — routed evals are impossible by construction.
1---2name: audit-activation3description: Audits sub-agent activation from explicit consumer telemetry roots. It separates attempted, started, completed, and unattributed events, reports coverage, and identifies observed zero starts without making an estate-wide dormancy claim. Use when specialized agents feel under-used, before pruning the catalog, or after wiring new agent spawn paths.4---5
6# audit-activation
7
8Reports whether OrchestKit's specialized sub-agents are actually being activated, from **real spawn telemetry** — not vibes. Scope is agent *activation*, not skill *quality*.
9
10It answers: "Do my specialized agents get spawned, or does the model default to generic Explore/general-purpose?" Read-only — it never edits agents.
11
12> **Why this exists:** a 2026-06 audit found only ~14% of agent spawns hit the 36-agent catalog vs ~74% generic, with 17/36 agents dormant — and that agents fire ~1:1 with how often a high-traffic skill names them via `subagent_type=`. Description rewrites ("use proactively") were A/B-tested and gave Δ0, so this skill measures **wiring + usage**, not description prose. See `docs/feat--activation-audit/`.
13
14## Quick Reference
15
16| Category | File | Impact | When to Use |
17|----------|------|--------|-------------|
18| Activation Checks | `rules/activation-checks.md` | HIGH | What to compute per agent |
19| Classification | `rules/activation-status.md` | HIGH | fires / mis-triggered / niche / dead buckets |
20| Output Format | `references/output-format.md` | MEDIUM | Report layout + the spawn-split summary |
21
22## CRITICAL: Task Management is MANDATORY (CC 2.1.16)
23
24```python
25TaskCreate(subject="Audit activation: agent spawn analysis",
26 description="Computing generic-vs-specialist split + dormancy from spawn telemetry",
27 activeForm="Auditing agent activation")
28TaskCreate(subject="Read spawn telemetry", activeForm="Reading subagent-spawns.jsonl")
29TaskCreate(subject="Compute split + observed zero starts", activeForm="Computing activation events and coverage")
30TaskCreate(subject="Classify & render", activeForm="Classifying agents and rendering report")
31TaskUpdate(taskId="3", addBlockedBy=["2"])
32TaskUpdate(taskId="4", addBlockedBy=["3"])
33```
34
35## Workflow
36
371. **Run the script FIRST** — every audit starts by running (or, when execution is impossible, explicitly referencing) the deterministic collector:
38
39 ```bash
40 bash "${CLAUDE_SKILL_DIR}/scripts/run-activation-audit.sh" \
41 --telemetry-root /path/to/consumer-project \
42 --stop-feed "$HOME/.claude/analytics/agent-usage.jsonl" \
43 --days 30 --json
44 ```
45
46 It resolves the catalog beside its source or installed script, while every consumer telemetry root is explicit. It reports attempts, starts, and stop-feed completions separately. Never eyeball JSONL by hand when the script exists.
472. **Inventory** — let the script resolve its own source or installed `agents/` catalog. Do not point it at a consumer project's agent directory.
483. **Read telemetry** — `.claude/logs/subagent-spawns.jsonl` carries pretool intent and start events. Triangulate it with the restored global `~/.claude/analytics/agent-usage.jsonl` stop feed when available. The streams have no stable common event ID, so never add them together or infer unique spawns.
494. **Compute** — all checks from `Read("rules/activation-checks.md")`; the Report Contract below lists the mandatory ones.
505. **Classify** — bucket every agent using the Four Buckets below (full procedure: `Read("rules/activation-status.md")`).
516. **Render** — output per `Read("references/output-format.md")`, satisfying the Report Contract.
52
53## Report Contract (every audit MUST include all six)
54
551. **Data-source line (first line of the report)** — name `scripts/run-activation-audit.sh`, every explicit consumer `.claude/logs/subagent-spawns.jsonl` root, the UTC window, and the spawn coverage state (`observed`, `missing`, `empty`, or `partial`). State whether `~/.claude/analytics/agent-usage.jsonl` was supplied as a stop feed. A report that presents numbers without these coverage facts is invalid.
562. **Spawn split** — generic (Explore/general-purpose/Plan) vs **ork-catalog** vs other-plugin, as **counts AND percentages** (e.g. "412 generic (74%) / 78 ork-catalog (14%) / 67 other (12%)"). Never percentages alone.
573. **Concentration** — the **top-5 agents' share of all catalog spawns** as a percentage (e.g. "top-5 = 81% of catalog spawns"), plus the top-5 list with fire counts.
584. **Observed-zero-start list** — only when spawn coverage is `observed`, enumerate every catalog agent with zero start events in the supplied roots, one per line with its reference count and bucket. This is not an estate-wide or lifetime dormancy claim. Do not render the list when coverage is missing, empty, or partial.
595. **Window caveat** — state that observed-zero-start names are absent only from the supplied roots and UTC window. They are a routing signal, not proof of zero estate-wide or lifetime use.
606. **Four-bucket classification table** (below) + **fix recommendations** — wiring changes only (see Hard Rules).
61
62## Classification: the Four Buckets (`rules/activation-status.md`)
63
64Bucket by **reference counts**, never by description quality. For each observed-zero-start agent, count its real spawn references: `grep -rc "subagent_type=ork:<name>" src/skills/` (plus `agent:`/team-map mentions in `src/agents/`). **Show the evidence**: cite that grep command in the report and put each agent's ref-count (with an example source file, e.g. `src/skills/cover/SKILL.md`) in the classification table. The telemetry observation proves only starts in supplied roots; the grep proves wiring. Render classification only when coverage is `observed`.
65
66| Bucket | Condition | Action |
67|--------|-----------|--------|
68| **ACTIVE** | fired ≥1× in the telemetry window | none — it works |
69| **MIS-TRIGGERED** | observed zero starts, has references but no real `subagent_type=` spawn from a busy skill | wire a spawn from a high-traffic skill |
70| **NICHE** | observed zero starts, rare-by-nature domain (design, multimodal, perf, IaC, security-LLM) AND wired somewhere | keep - low use expected |
71| **DEAD** | observed zero starts AND **zero references in ANY skill or agent** | prune candidate |
72
73**DEAD requires zero references anywhere** — an agent with even one reference is never DEAD; it is mis-triggered or niche. Only DEAD agents are prune candidates.
74
75## Hard Rules
76
77- **Never recommend rewriting agent descriptions.** A/B-tested: description rewrites gave Δ0 routing benefit. Dormancy is a **skill-wiring problem** — the fix is always adding a `subagent_type=ork:<name>` spawn from a high-traffic skill (e.g. implement, cover, review-pr), never prose changes.
78- **Classify by reference count, not description quality.** The grep above is the evidence; "the description is vague" is not.
79- **State the telemetry window, supplied roots, and coverage state in EVERY answer** — including classification-only or prune-decision answers. Observed-zero-start is never an estate-wide dormancy claim.
80- **Keep event semantics separate.** Pretool is attempted intent, `source:start` is started activation, and `agent-usage.jsonl` is a stop/completion observation. Cite all supplied sources and never sum them into a unique-spawn claim.
81- **Answer with the report itself, never with only a description of the process.** Even when you cannot execute the script in the current context, render the full contract-format report (state which numbers are from the latest available run vs illustrative).
82- Read-only: this skill never edits agents.
83
84## Example report shape (illustrative numbers — imitate the FORM exactly)
85
86```markdown
87Data: scripts/run-activation-audit.sh over /consumer/.claude/logs/subagent-spawns.jsonl (window: 2026-05-23T00:00:00Z to 2026-07-14T00:00:00Z; coverage: observed)
88Stop feed: ~/.claude/analytics/agent-usage.jsonl supplied separately; it is not joined to starts.
89
90## Spawn split
91Total 560 spawns: 412 generic (74%) / 78 ork-catalog (14%) / 70 other-plugin (12%)
92
93## Concentration
94Top-5 = 63/78 catalog spawns (81%): web-research-analyst 26, code-quality-reviewer 21,
95test-generator 8, debug-investigator 5, backend-system-architect 3
96
97## Observed zero-start agents (6 of 20 in the supplied roots, refs via grep -rc "subagent_type=ork:<name>" src/skills/)
98- emulate-engineer (3 refs) -> MIS-TRIGGERED
99- expect-agent (3 refs) -> MIS-TRIGGERED
100- design-system-architect (2 refs) -> NICHE (rare domain, wired)
101- multimodal-specialist (1 ref) -> NICHE (rare domain, wired)
102- infrastructure-architect (2 refs) -> NICHE (rare domain, wired)
103- event-driven-architect (1 ref) -> MIS-TRIGGERED
104
105Caveat: observed zero starts are absent only from the supplied telemetry roots and
106window. They are not proof of zero estate-wide or lifetime use.
107
108## Buckets (rules/activation-status.md)
109ACTIVE (14) | MIS-TRIGGERED (3) | NICHE (3) | DEAD (0 — none: every agent has >=1 reference)
110
111## Fixes (wiring only — never description rewrites)
112- emulate-engineer: add subagent_type=ork:emulate-engineer spawn from cover (src/skills/cover/SKILL.md)
113- expect-agent: add subagent_type=ork:expect-agent spawn from review-pr
114```
115
116## Chain
117
118After this audit, run the deeper experiment if you suspect descriptions: `docs/feat--activation-audit/agent-routing-experiment.mjs` (isolated A/B selection harness).
119
120## Related Skills
121
122- `telemetry-inspect` — validates the telemetry data-plane this skill reads from
123- `analytics` — raw usage queries across projects
124- `doctor` — broader plugin health (manifests, hooks, memory budget)
125
126> Eval note: this skill is direct-only (`disable-model-invocation: true`), so quality evals run in TIER-1 unit mode (`--force-skill`) — routed evals are impossible by construction.
yonatangross