guerrilla2799 Agent Observability
Instrument before scaling, not after. A system with no telemetry has no maintenance, and a system with no maintenance degrades on a predictable schedule.
When to use
- Before any agent goes from one user to several
- Nobody can answer how something is performing
- Quality complaints arrive with no data behind them
- Before a scale-up decision
Inputs
- Reads:
workspace/agents/<name>/brief.mdfor the success threshold,workspace/evals/log/eval-log.jsonl - Needs from user: where logs can be written, and who reads the weekly review
Workflow
1. Log the run, not just the output
Every run writes one line. Append-only.
{"ts":"<iso>", "agent":"<name>", "version":"<semver>", "user":"<id>",
"input_summary":"<short>", "tools_called":[...], "tool_spend":n,
"duration_s":n, "guardrail_trips":[...], "eval_verdict":"<if judged>",
"outcome":"kept|edited|discarded"}
outcome is the field that matters and the one that requires a human to supply it. Everything else is automatic and less informative. Whether the user kept the output, edited it, or threw it away is the only direct measure of usefulness, and it has to be captured at the moment of use.
Make it a single keystroke. A three-field feedback form gets filled in for a week and then never again.
2. Track the six numbers
| Metric | Source | Healthy | What a bad reading means |
|---|---|---|---|
| Weekly active users | Run log | Over 60% of intended | Rollout failure. See agent-rollout |
| Runs per active user | Run log | Over 5 per week | It is a novelty, not a tool |
| Correction rate | outcome field |
Under 30% | It is a draft generator. Price it as one |
| Cost per run | tool_spend plus inference |
Track the trend | A climbing trend usually means context bloat |
| Guardrail trip rate | guardrail_trips |
2 to 10% | 0% means they are decorative. Over 20% means wrong scope |
| Eval pass rate | Eval log | 40 to 65% first pass | Over 85% and the rubric is soft. See eval-calibration |
3. Alert on shape, not just on failure
The dangerous failures do not throw errors.
| Alert on | Because |
|---|---|
| Correction rate rising over 3 weeks | Usually a stale context portfolio, not a worse model |
| Cost per run rising with flat usage | Context bloat or a retry loop |
| Guardrail trip rate falling to zero | Either nobody is using it or a check silently broke |
| Output field distribution shifting over 20% | Something upstream changed. This is the silent-sync failure |
| Log volume dropping | The logging broke, and you are now blind |
The last one is the one people forget. A monitoring system that stops reporting looks exactly like a healthy quiet week.
4. Diagnose degradation in the right order
When quality drops, check in this sequence. It runs cheapest first and catches the common cases early.
1. Did the inputs change? data freshness, source changes, a broken sync
2. Did the context go stale? positioning, proof points, plays out of date
3. Did the spec drift? run eval-calibration
4. Did usage change? people using it for something it was not scoped for
5. Did the model or tooling change? last, because it is the least likely and the least fixable
Teams reliably start at step 5 and skip steps 1 and 2, which are the actual cause most of the time.
5. Run the weekly review
Fifteen minutes, one page, a named reader.
AGENT TELEMETRY: week of <date>
<agent> WAU n/n runs n correction n% cost/run n trips n%
<agent> ...
Alerts: <anything tripped>
Corrections made: <written to the context portfolio, not to chat>
Decision: <continue | fix <specific thing> | retire>
Corrections get written to the context portfolio. A correction made in a review meeting and not written to a file is lost, and the same correction will be made again next month.
6. Retire on the kill criterion
The brief named a kill criterion. At the review date, apply it. An unused agent is not neutral, because its description competes for selection with the ones that work.
Output
- Writes:
workspace/evals/telemetry-spec.md,workspace/evals/log/runs.jsonl, the weekly review - Prints: the six numbers, any alerts, and the decision
Rules & quality bar
- Instrument before scaling. Not after the first complaint
- The
outcomefield is captured from the human, in one keystroke - Append-only logs, including failures
- Alert on shape and on silence, not only on errors
- Diagnose inputs and context before blaming the model. In that order
- Corrections write to the context portfolio
- A named reader for the weekly review, or it stops happening
- Apply the kill criterion on schedule
Related skills
- Requires:
agent-rolloutfor the intended-user denominator - Pairs with:
eval-loop,eval-calibration,credit-and-cost-control - See also:
docs/metrics.md,docs/field-notes.mdstory 11